What temperature is recommended when doing RAG?

Key Takeaways

language models kind of break see the world in terms of tokens which is kind of like the minimum unit of text you can put in a language model to get it to get predicted next token it might not be a whole word it might be like a comma it might be even a space it might just be the word the but essentially these models what they do is they take in these tokens and they take into account the position of all the tokens in the input and they they use all the information they're given to predict the next token which is why they're so powerful when we talk about temperature uh we're talking about basically this probability distribution that the model is using to predict essentially there's this concept of I could pick the most likely next token or I could not so temperature is one of the many tools you have to kind of change how these models are using the probability distribution to pick the next token um when it comes to rag unfortunately the answer is it depends in I feel like most of the chat your data scenarios I have personally seen people want you want to kind of go with where the prob distribution is pushing you if you're looking for a more creative response where it's like oh I want you to like rephrase how this document says this uh lower temperature or sorry higher temperatures might be a better choice uh it all depends on how this probability distribution you you kind of uh is really something you have to play with to get a sense of how it works for your particular data