Let's reproduce GPT-2 (124M)

Andrej Karpathy · Advanced ·🧠 Large Language Models ·2y ago

Key Takeaways

The video demonstrates how to reproduce the GPT-2 (124M) model from scratch, covering the process of building the GPT-2 network, optimizing its training, setting up the training run, and generating results. The video uses tools such as PyTorch, Hugging Face, and Lambda GPU Cloud.

Original Description

We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Links: - build-nanogpt GitHub repo, with all the changes in this video as individual commits: https://github.com/karpathy/build-nanogpt - nanoGPT repo: https://github.com/karpathy/nanoGPT - llm.c repo: https://github.com/karpathy/llm.c - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - Attention is All You Need paper: https://arxiv.org/abs/1706.03762 - OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165 - OpenAI GPT-2 paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf- The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com Chapters: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 paramet
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from Andrej Karpathy · Andrej Karpathy · 15 of 17

← Previous Next →

This video teaches how to reproduce the GPT-2 (124M) model from scratch, covering the process of building the network, optimizing training, and generating results. The video provides a comprehensive guide to building and training large language models.

Key Takeaways
  1. Build the GPT-2 network using PyTorch
  2. Load pre-trained model parameters from Hugging Face
  3. Implement the forward pass to get logits
  4. Sample text using the trained model
  5. Train the model on a custom dataset
  6. Optimize hyperparameters for better performance
💡 The video highlights the importance of optimizing the training process for large language models, including the use of batched data and custom data loaders.

Related Reads

📰
Building Multimodal SuperAgents: Integrating Speech, OCR, and Translation with iFly-Skills
Learn to build multimodal SuperAgents that integrate speech, OCR, and translation using iFly-Skills, enabling AI to interact with the physical world
Dev.to AI
 📰
I built a Google Sheets MCP server so Claude reads my sheets
Turn a Google Sheet into a read-only MCP server that Claude can query using plain English, no coding required
Dev.to · Jay from PasteSheet
 📰
OpenAI scored an own goal with HuggingFace attack, showing how open Chinese models are winning
OpenAI's criticism of Hugging Face's Chinese model highlights the vulnerability of closed models and the rising competitiveness of open Chinese models
The Register
 📰
citesure 0.5.17: soft-check ORCID iDs before you ship
Learn to soft-check ORCID iDs in bibliographies using citesure 0.5.17 to ensure data accuracy
Dev.to · SybilGambleyyu

Chapters (13)

intro: Let’s reproduce GPT-2 (124M)
3:39 exploring the GPT-2 (124M) OpenAI checkpoint
13:47 SECTION 1: implementing the GPT-2 nn.Module
28:08 loading the huggingface/GPT-2 parameters
31:00 implementing the forward pass to get logits
33:31 sampling init, prefix tokens, tokenization
37:02 sampling loop
41:47 sample, auto-detect the device
45:50 let’s train: data batches (B,T) → logits (B,T,C)
52:53 cross entropy loss
56:42 optimization loop: overfit a single batch
1:02:00 data loader lite
1:06:14 paramet
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →