Differential Privacy in 30 seconds

AI Coffee Break · Advanced ·🧠 Large Language Models ·8mo ago

Key Takeaways

Differential privacy in large language models is achieved through techniques such as gradient clipping and adding Gaussian noise, as seen in Google's VaultGemma, the first open-weight LLM trained from scratch with differential privacy.

Full Transcript

Step one, take a big batch of sequences and pass them through the model's weights to get an output. In this example, we have a batch of two documents. For each sequence, compute the gradients which capture how the model should change to better predict the next token, but clip the norm of the gradient so no single sequence gets to dominate. Then compute the average gradient over the entire batch. And step two, add a dose of gshian noise onto those clipped gradients. The noise is tuned so single examples can't be detected but repeated patterns still show up. Step three, update the weights with that noisy average. Repeat across a trillion tokens.

Original Description

Full video: 🎥 https://youtu.be/UwX5zzjwb_g Large language models often memorize what they see — even a single phone number or address can stick forever in their weights. Google’s new VaultGemma changes that: it’s the first open-weight LLM trained from scratch with differential privacy, meaning secrets seen seldomly during training leaves no trace. 👉 In this video, we explain Differential Privacy through the concrete example of VaultGemma — how it works, why it matters, and what it means for the future of trustworthy AI. AI Coffee Break Merch! 🛍️ https://aicoffeebreak.creator-spring.com/ Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏 Vignesh Valliappan, Ivan Janov, Sunny Dhiana, Andy Ma ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak Join this channel as a Bean Member to get access to perks: https://www.youtube.com/channel/UCobqgqE4i5Kf7wrxRxhToQA/join ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter / X: https://twitter.com/AICoffeeBreak LinkedIn: https://www.linkedin.com/in/letitia-parcalabescu/ Threads: https://www.threads.net/@ai.coffee.break Bluesky: https://bsky.app/profile/aicoffeebreak.bsky.social Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak Substack: https://aicoffeebreakwl.substack.com/ Web: https://explanationmark.de/letitia https://aicoffeebreak.com #AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​ Video editing: Nils Trost
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from AI Coffee Break with Letitia · AI Coffee Break with Letitia · 0 of 60

← Previous Next →
1 AI Coffee Break - Channel Trailer
AI Coffee Break - Channel Trailer
AI Coffee Break with Letitia
 2 How to check if a neural network has learned a specific phenomenon?
How to check if a neural network has learned a specific phenomenon?
AI Coffee Break with Letitia
 3 A brief history of the Transformer architecture in NLP
A brief history of the Transformer architecture in NLP
AI Coffee Break with Letitia
 4 Our paper at CVPR 2020 - MUL Workshop and ACL 2020 - ALVR Workshop
Our paper at CVPR 2020 - MUL Workshop and ACL 2020 - ALVR Workshop
AI Coffee Break with Letitia
 5 The Transformer neural network architecture EXPLAINED. “Attention is all you need”
The Transformer neural network architecture EXPLAINED. “Attention is all you need”
AI Coffee Break with Letitia
 6 Transformer combining Vision and Language? ViLBERT - NLP meets Computer Vision
Transformer combining Vision and Language? ViLBERT - NLP meets Computer Vision
AI Coffee Break with Letitia
 7 Pre-training of BERT-based Transformer architectures explained – language and vision!
Pre-training of BERT-based Transformer architectures explained – language and vision!
AI Coffee Break with Letitia
 8 GPT-3 explained with examples. Possibilities, and implications.
GPT-3 explained with examples. Possibilities, and implications.
AI Coffee Break with Letitia
 9 Adversarial Machine Learning explained! | With examples.
Adversarial Machine Learning explained! | With examples.
AI Coffee Break with Letitia
 10 BERTology meets Biology | Solving biological problems with Transformers
BERTology meets Biology | Solving biological problems with Transformers
AI Coffee Break with Letitia
 11 Can a neural network tell if an image is mirrored? – Visual Chirality
Can a neural network tell if an image is mirrored? – Visual Chirality
AI Coffee Break with Letitia
 12 The ultimate intro to Graph Neural Networks. Maybe.
The ultimate intro to Graph Neural Networks. Maybe.
AI Coffee Break with Letitia
 13 Can language models understand? Bender and Koller argument.
Can language models understand? Bender and Koller argument.
AI Coffee Break with Letitia
 14 GANs explained | Generative Adversarial Networks video with showcase!
GANs explained | Generative Adversarial Networks video with showcase!
AI Coffee Break with Letitia
 15 What nobody tells you about MULTIMODAL Machine Learning! 🙊 THE definition.
What nobody tells you about MULTIMODAL Machine Learning! 🙊 THE definition.
AI Coffee Break with Letitia
 16 Multimodal Machine Learning models do not work. Here is why. Part 1/2 – The SYMPTOMS
Multimodal Machine Learning models do not work. Here is why. Part 1/2 – The SYMPTOMS
AI Coffee Break with Letitia
 17 Why Multimodal Machine Learning models do not work. Part 2/2 – The CAUSES
Why Multimodal Machine Learning models do not work. Part 2/2 – The CAUSES
AI Coffee Break with Letitia
 18 An image is worth 16x16 words: ViT | Vision Transformer explained
An image is worth 16x16 words: ViT | Vision Transformer explained
AI Coffee Break with Letitia
 19 AI understanding language!? A roadmap to natural language understanding.
AI understanding language!? A roadmap to natural language understanding.
AI Coffee Break with Letitia
 20 "What Can We Do to Improve Peer Review in NLP?" 👀
"What Can We Do to Improve Peer Review in NLP?" 👀
AI Coffee Break with Letitia
 21 The curse of dimensionality. Or is it a blessing?
The curse of dimensionality. Or is it a blessing?
AI Coffee Break with Letitia
 22 PCA explained with intuition, a little math and code
PCA explained with intuition, a little math and code
AI Coffee Break with Letitia
 23 Data-efficient Image Transformers EXPLAINED! Facebook AI's DeiT paper
Data-efficient Image Transformers EXPLAINED! Facebook AI's DeiT paper
AI Coffee Break with Letitia
 24 OpenAI's DALL-E explained. How GPT-3 creates images from descriptions.
OpenAI's DALL-E explained. How GPT-3 creates images from descriptions.
AI Coffee Break with Letitia
 25 Leaking training data from GPT-2. How is this possible?
Leaking training data from GPT-2. How is this possible?
AI Coffee Break with Letitia
 26 OpenAI’s CLIP explained! | Examples, links to code and pretrained model
OpenAI’s CLIP explained! | Examples, links to code and pretrained model
AI Coffee Break with Letitia
 27 Transformers can do both images and text. Here is why.
Transformers can do both images and text. Here is why.
AI Coffee Break with Letitia
 28 UMAP explained | The best dimensionality reduction?
UMAP explained | The best dimensionality reduction?
AI Coffee Break with Letitia
 29 NVIDIA Jarvis (now NVIDIA Riva) meets Ms. Coffee Bean
NVIDIA Jarvis (now NVIDIA Riva) meets Ms. Coffee Bean
AI Coffee Break with Letitia
 30 Transformer in Transformer: Paper explained and visualized | TNT
Transformer in Transformer: Paper explained and visualized | TNT
AI Coffee Break with Letitia
 31 [RANT] Adversarial attack on OpenAI’s CLIP? Are we the fools or the foolers?
[RANT] Adversarial attack on OpenAI’s CLIP? Are we the fools or the foolers?
AI Coffee Break with Letitia
 32 Pattern Exploiting Training explained! | PET, iPET, ADAPET
Pattern Exploiting Training explained! | PET, iPET, ADAPET
AI Coffee Break with Letitia
 33 Deep Learning for Symbolic Mathematics!? | Paper EXPLAINED
Deep Learning for Symbolic Mathematics!? | Paper EXPLAINED
AI Coffee Break with Letitia
 34 FNet: Mixing Tokens with Fourier Transforms – Paper Explained
FNet: Mixing Tokens with Fourier Transforms – Paper Explained
AI Coffee Break with Letitia
 35 Are Pre-trained Convolutions Better than Pre-trained Transformers? – Paper Explained
Are Pre-trained Convolutions Better than Pre-trained Transformers? – Paper Explained
AI Coffee Break with Letitia
 36 "Please Commit More Blatant Academic Fraud" – A fellow PhD student's response.
"Please Commit More Blatant Academic Fraud" – A fellow PhD student's response.
AI Coffee Break with Letitia
 37 Scaling Vision Transformers? How much data can a transformer get? #Shorts
Scaling Vision Transformers? How much data can a transformer get? #Shorts
AI Coffee Break with Letitia
 38 How cross-modal are vision and language models really? 👀 Seeing past words. [Own work]
How cross-modal are vision and language models really? 👀 Seeing past words. [Own work]
AI Coffee Break with Letitia
 39 Charformer: Fast Character Transformers via Gradient-based Subword Tokenization +Tokenizer explained
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization +Tokenizer explained
AI Coffee Break with Letitia
 40 Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.
Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.
AI Coffee Break with Letitia
 41 Adding vs. concatenating positional embeddings & Learned positional encodings
Adding vs. concatenating positional embeddings & Learned positional encodings
AI Coffee Break with Letitia
 42 Self-Attention with Relative Position Representations – Paper explained
Self-Attention with Relative Position Representations – Paper explained
AI Coffee Break with Letitia
 43 Saddle points vs. local minima in high dimensional spaces | ❓ #AICoffeeBreakQuiz #Shorts
Saddle points vs. local minima in high dimensional spaces | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 44 What is the model identifiability problem? | Explained in 60 seconds! | ❓ #AICoffeeBreakQuiz #Shorts
What is the model identifiability problem? | Explained in 60 seconds! | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 45 Data leakage during data preparation? | Using AntiPatterns to avoid MLOps Mistakes
Data leakage during data preparation? | Using AntiPatterns to avoid MLOps Mistakes
AI Coffee Break with Letitia
 46 Is today's AI smarter than YOU? #Shorts
Is today's AI smarter than YOU? #Shorts
AI Coffee Break with Letitia
 47 Convolution vs Cross-Correlation. How most CNNs do not compute convolutions. | ❓ #Shorts
Convolution vs Cross-Correlation. How most CNNs do not compute convolutions. | ❓ #Shorts
AI Coffee Break with Letitia
 48 Why do we care about cross-correlations vs convolutions | ❓ #AICoffeeBreakQuiz #Shorts
Why do we care about cross-correlations vs convolutions | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 49 The convolution is not shift invariant. | Invariance vs Equivariance | ❓ #AICoffeeBreakQuiz #Shorts
The convolution is not shift invariant. | Invariance vs Equivariance | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 50 How to increase the receptive field in CNNs? | #AICoffeeBreakQuiz #Shorts
How to increase the receptive field in CNNs? | #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 51 What is tokenization and how does it work? Tokenizers explained.
What is tokenization and how does it work? Tokenizers explained.
AI Coffee Break with Letitia
 52 Foundation Models | On the opportunities and risks of calling pre-trained models “Foundation Models”
Foundation Models | On the opportunities and risks of calling pre-trained models “Foundation Models”
AI Coffee Break with Letitia
 53 How modern search engines work – Vector databases explained! | Weaviate open-source
How modern search engines work – Vector databases explained! | Weaviate open-source
AI Coffee Break with Letitia
 54 Eyes tell all: How to tell that an AI generated a face?
Eyes tell all: How to tell that an AI generated a face?
AI Coffee Break with Letitia
 55 Swin Transformer paper animated and explained
Swin Transformer paper animated and explained
AI Coffee Break with Letitia
 56 Data BAD | What Will it Take to Fix Benchmarking for NLU?
Data BAD | What Will it Take to Fix Benchmarking for NLU?
AI Coffee Break with Letitia
 57 SimVLM explained | What the paper doesn’t tell you
SimVLM explained | What the paper doesn’t tell you
AI Coffee Break with Letitia
 58 Generalization – Interpolation – Extrapolation in Machine Learning: Which is it now!?
Generalization – Interpolation – Extrapolation in Machine Learning: Which is it now!?
AI Coffee Break with Letitia
 59 Do Transformers process sequences of FIXED or of VARIABLE length? | #AICoffeeBreakQuiz
Do Transformers process sequences of FIXED or of VARIABLE length? | #AICoffeeBreakQuiz
AI Coffee Break with Letitia
 60 The efficiency misnomer | Size does not matter | What does the number of parameters mean in a model?
The efficiency misnomer | Size does not matter | What does the number of parameters mean in a model?
AI Coffee Break with Letitia

This video explains how differential privacy can be applied to large language models to prevent memorization of sensitive information, using techniques such as gradient clipping and Gaussian noise. The VaultGemma model is introduced as an example of a successfully trained open-weight LLM with differential privacy. By understanding these concepts, viewers can implement differential privacy in their own LLMs and improve the security of their models.

Key Takeaways
  1. Take a big batch of sequences and pass them through the model's weights to get an output
  2. Compute the gradients for each sequence and clip the norm of the gradient
  3. Compute the average gradient over the entire batch
  4. Add Gaussian noise to the clipped gradients
  5. Update the weights with the noisy average
  6. Repeat the process across a large dataset
💡 Differential privacy can be achieved in LLMs by adding noise to the gradients during training, preventing the model from memorizing individual sequences while still learning from repeated patterns.

Related Reads

📰
I compared the real cost of running LLMs on AWS - here's when each option makes sense
Learn when to use each AWS option for running LLMs in production and understand their cost implications
Dev.to · Jerzy Kopaczewski
 📰
Building a Character-Level Bigram Language Model from Scratch with PyTorch
Learn to build a basic character-level bigram language model from scratch using PyTorch, understanding the fundamentals of neural language modeling
Dev.to · Mohamed Heni
 📰
Running NVIDIA Nemotron 3.5 ASR Locally with parakeet.cpp (and how it beat Whisper on my laptop)
Run NVIDIA Nemotron 3.5 ASR locally for offline speech-to-text capabilities without relying on cloud services or incurring API bills
Medium · LLM
 📰
When Does a Prompt Become an Undocumented Program?
Learn to identify when a prompt becomes an undocumented program and why it matters for effective AI integration in analyst work
Dev.to · Yura Solovey
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →