LLM Foundations: Understanding Tokenization & Training: Chapter 4

Key Takeaways

[Music] welcome to module 2 in this module we want to understand how large language models work but first let's check out some use cases that llms enable llms can be used to generate text like marketing copy or emails they can answer questions translate documents and determine the sentiment of a text llms can summarize long documents they can act as personal assistants or chatbots we can use them to query tabular data interact with API or even evaluate other language models but what happens behind the scenes understanding llm architecture isn't necessary for building applications it's like driving a car you don't need to know how the engine works to drive still some technical details can be helpful looking at gp4 technical report we can read that gp4 one of the most known llms is a Transformer based model pre-trained to predict the next token in a document we won't dive and and try to understand the Transformer architecture that's not necessary for building Alm applications but we want to we want to focus on the second part of the statement which is predicting the next token in a document so here's how this works we start with some input text in our case weights and biases is then we tokenize the text we need to split it into tokens that are represented by numbers that we fit into the black box which is the llm then as an output of the llm we have a distribution of probabilities over the entire vocabulary all of the tokens that we have available for our model and each of these tokens comes with a probability that it comes as a next token in the sequence and based on those probabilities we pick we sample uh one of the tokens to follow to continue with the sequence in this case we select the token the because it has a high output probability then we append this token to our input sequence and we repeat the process we tokenize it we fit it into the llm and again we get a distribution of probabilities across our vocabulary all of the tokens and again we pick a token with high probability in this case let's pick machine and finally uh we again repeat this whole process and we sample the token learning and if we continue with this process we can predict we can sample the text weights and biases is the machine learning platform companies like open AI C here Mosaic or meta have already trained models for us and we use them behind apis which means we do not need to train these models to use them in our applications however knowing how they were trained can provide useful insights there are two main steps in training llms the first is pre-training where the model learns from a massive data set with sources like the entire internet such as common C C4 uh GitHub Wikipedia book books archive which are academic papers and Stock Exchange which is a set of questions and answers this pre-training data set has been published by meta uh that trained Lama model we don't know exactly the pre-training data set used for training GPT 4 but we can imagine it must have been something similar in this case in pre-training a model that has gone through this phase is pretty good in predicting texts such as found in this data set on the internet uh on on GitHub on Wikipedia and so on but this may not be enough we actually want this model to follow our instructions to respond to our questions and this is where the Second Step which is supervised instruction tuning uh can be helpful in this step the model is further trained with expert generated question answer Pairs and this helps align the model with user expectations and follow instructions some llms like gp4 undergo an additional phase reinforcement learning from Human feedback here the model is trained to optimize for higher quality answers preferred by human judges understanding these training phases can be helpful it can give us intuitions for example how to formulate a prompt in order to get the expected answer the expected output from the model in the next video we'll experiment with this concept in Jupiter notebook with code