Why Large Language Models (LLMs) Output Differs

Key Takeaways

Large language models by design are usually what is called non-deterministic. So what this means is the same input will not always give you the same output. So if you think mathematically like 2 + 2, it's always going to equal 4 or like 2 + x = y. I mean if you keep putting in 2 to that equation, then y is always going to be four. In an LLM, by design, they're usually not deterministic. So what does this mean? If you put the same input in, even if you just say hi to it many times, you're going to get either, you know, slightly varying to largely varying. And this depends on how the LLM is set up and configured. And in particular, there's two main variables that impact this called the temperature and top P. If you look at how an LLM predicts uh text or tokens, it's tokens, but you can think of them as just words for easy representation. the output representation or the output layer of the LLM is going to be a probability for all the tokens in its vocabulary. So everything it knows all of those tokens as like words or chunks of words or vocabulary or even special symbols like this is the end of the sequence uh and it's going to be a probability distribution for all of those. So the highest probability token um if we were being deterministic and using what's called the greedy decoding, we would just always take the top one. But this isn't as interesting and doesn't really mimic as well as people talk because you know we have slight variations and how we talk and you know if you're just always getting the same thing uh it's not as good to interact with and doesn't feel like a human. So what the temperature does is it actually um changes the probability distribution to flatten that probability distribution and then we just use like uh statistics to pick from that. So we won't always pick the top one but we'll use a weighted statistics approach and that's what the temperature does. So the higher the temperature then the more random the outputs can um seem and then that top P that is the percentage of probability distribution that we'll select from. So for example we use a top P of 0.9 we would use all of the tokens that their probability adds up to 0.9 and that just lets us throw away like the low stuff that's like oh that's like 0.1%. We don't want any chance of that getting it because it says it's junk. So, let's just take like the top pile of them and then do a weighted approach and we'll take from that and then it's more likely that we'll take the top ones, but you know, we could throw in a few other words and then with how um LLMs work then they just keep feeding those tokens back in, right? It's called autoagression. So that randomness starts to add up especially for longer outputs where yeah you can get very differed outputs.