Deep Dive into LLM Sampling Techniques: Chapter 7

Key Takeaways

[Music] okay now that we understand how tokens work let's move on to sampling we discussed different sampling methods and the first one we'll experiment with is temperature we can provide temperature as a as a parameter to open AI completion function uh we use this to generate um to generate a response to some text so in this case we will use our text D Vinci 003 model our prompt is going to be say something about weights and biases and then we will also limit the output to 50 tokens and we will set temperature as a as a hyper parameter and then return the text out of the the API response so let's uh run this function and then let's generate text for different temperature values ranging from zero to two and the intuition is like temperature zero is is gritty the coding it should sample the most probable tokens in a sequence and in this case the model says that weights and biases is an amazing tool for tracking and analyzing machine learning experiments it provides powerful visualizations and insights into model performance enabling data scientists to quickly identify areas of improvement and optimize their models I'm quite happy with this generation it feels like um the model knows um the the the value proposition of weights and biases and we can see how this changes as we as we increase temperature parameter and you can see now we're getting different Generations so with temperature one uh weight Andes is a powerful tool for managing and monitoring machine learning projects it provides real-time insights into the performance of your models and presents a wide range of visualizations to ensure that data is is easy to interpret again quite happy with this generation what happens if we further increase the temperature so with temperature value two um the model says uh weights and biases is an awesome analytics and visualization platform designed specifically for everything from model interviews to share neural Nets um and you can see like here actually this uh becomes a bit gibberish and and it probably is too high of a value of a temp which means that tokens with low probabilities get sampled and this results in text which isn't really uh that meaningful so here um probably temperature like between zero and one is something that we want to go to go for uh using this model we also discussed the top P sampling which is another sampling method uh open AI um encourage us to pick one of those two parameters and not play with both um so let's now try to to experiment with top P sampling parameter and again we use the same model we'll use the same prompt and now we'll uh generate text with different top values and again we'll go with a range from 0 01 to one in the case of the the lowest uh value of top p uh we will pick tokens with the uh with the highest probability and and we can see that we are hitting rate limit errors on the open AI um API so we're going to wait a moment and in the next um in the next section uh I will show you how to how to handle these errors these errors more gracefully with back off uh for now let's wait uh and when the rate limit uh stops hitting us we'll continue with this uh section okay it looks like the we were able to run the API now and we can see the different generations for different top P values and again with the lowest value of top P we are sampling a high probability text and we can read about weights and biases that it is an amazing tool for tracking and visualizing machine learning experiments it helps to keep track of all uh the different experiments you run and provides powerful visualizations to help you understand the results and again I'm quite happy with this result uh probably the uh 01 uh value is quite good as well now what happens if we increase top p uh so let's read for top P equals 1 weights and bies is an AI experimentation platform that helps teams understand track and collaborate on their machine learning projects it provides powerful visualizations realtime alerts and integrated workflows um and again this text is quite good and the reason is probably that we are still using the temperature parameter in the background so the the rare tokens are uh the the tokens with low probability are are very unlikely uh to be selected uh but again like this is probably a bit more diverse so experimenting with top P values is a different way of making sure that uh the model uh generates diverse and interesting um text on the other hand if you want to go for minimal risk and making sure like the highest probability taxt is generated then decreasing uh Tope or decreasing temperature might be a good a good choice