Sampling
An LLM always generates a list of next tokens with their probabilities. Sampling is a method to pick one token from this list based on the probabilities.
Most important to remember that sampling is an inference engine feature. Model has no idea about it. Top-k, top-p and temperature are all inference engine parameters to control the sampling process.
An LLM's output is always about predicting the next best token, or next word. Read also speculative decoding which is a technique to predict multiple tokens at once.
Example - Which is the capital of France?
The output is the next best word after the full text. So: Which is the capital of France? {$answer token}.
The whole input plus the generated tokens, word by word, is passed back to the LLM. This repeats until the LLM generates EOS as the next token.
Probability
The neural network output is always an array of float values. It holds one for each token in the LLM's vocabulary.
This value is then converted into probabilities.
Sampling Process
Sampling is a statistics method. It uses a subset of items to predict the whole.
For an LLM, sampling is done from the next token's probability distribution. It takes only a subset of tokens, based on temperature, Top-P, and Top-K.
-
Temperature: This parameter shapes how the LLM picks a token. Higher temperature gives more random output. At the lowest (0.0), it always returns the highest-probability value.
-
Top-P: This parameter sets how many tokens to consider. The LLM keeps the top tokens whose probabilities add up to the Top-P value.
-
Top-K: It just keeps the top-K tokens with the highest probability. The K in the parameter is for keep.
Greedy Decoding
With greedy decoding, the model always picks the highest-probability token. For example, temperature 0.0, Top-P 0.0, and Top-K 1.0.
LLMs only know how two things relate. They don't know cause and effect.