Exploring LLMs: Common Parameters in GPT-like Models - Top-k, Top-p, Temperature

Preface

The previous article introduced the environment setup and local deployment for several open-source LLMs. When using the ChatGPT interface or a locally deployed LLM, you often encounter these parameters. This article will briefly introduce them~

• temperature
• top_p
• top_k

About LLM

This was also mentioned in the previous article, but this time I came across a nice diagram.

A recent breakthrough in artificial intelligence (AI) is the introduction of language processing technologies that enable us to build more intelligent systems with a richer understanding of language than ever before. Large pre-trained Transformer language models, or simply large language models, vastly extend the capabilities of what systems are able to do with text.

While LLMs may seem magical, at their core, they are still a matter of probability. A neural network generates a set of candidate words based on the input text, drawing from its pre-trained model, and then selects the ones with the highest probability as the output. The three parameters mentioned above are all related to sampling—that is, how to choose an output from the candidate words.

temperature

Used to control the randomness of model output results; the higher this value, the greater the randomness. Generally, after inputting the same prompt multiple times, the model's output will be different each time.

• Set to 0, generates fixed output for each prompt
• Lower values, output is more concentrated and deterministic
• Higher values, output is more random (more creative 😃)

Generally, the longer the prompt and the clearer the description, the better the quality of the model's generated output and the higher the confidence. In this case, you can appropriately increase the temperature value. Conversely, if the prompt is very short and vague, setting a relatively high temperature value will make the model's output very unstable.

When in doubt, adjust the parameters; give it a try, and maybe you'll generate a good answer?

PS: The example provided by ChatGLM limits the range to between 0-1.

top_k & top_p

These two are also sampling parameters, but they use a different sampling method than temperature.

As mentioned earlier, before outputting, the model generates a bunch of tokens, which are ranked by quality.

For example, in the following image, given the input The name of that country is the, the model generates a bunch of tokens and then selects one for output based on different decoding strategies.

The decoding strategy here can be chosen as:

• greedy decoding: Always selects the token with the highest score. It's useful but has some drawbacks, as detailed below.
• top-k: Selects k tokens from the list as candidates and then samples based on their likelihood scores.
• top-p: The candidate list is dynamic, selecting candidates from the tokens based on a cumulative probability percentage.

top-k and top-p introduce randomness into the token selection process, giving other high-scoring tokens a chance to be chosen, unlike greedy decoding which always picks the highest score.

Greedy Decoding

The advantage is simplicity, the disadvantage is that it easily generates repetitive and circular content.

Greedy decoding is a reasonable strategy but has some drawbacks such as outputs with repetitive loops of text. Think of the suggestions in your smartphone's auto-suggest. When you continually pick the highest suggested word, it may devolve into repeated sentences.

top-k

The larger the setting, the more possibilities the generated content will have; the smaller the setting, the more fixed the generated content will be; when set to 1, the effect is the same as greedy decoding.

Changing the top-k parameter sets the size of the shortlist the model samples from as it outputs each token. Setting top-k to 1 gives us greedy decoding.

top-p

top-p, also known as Nucleus Sampling (核采样)

Unlike top-k, which selects the fixed top k tokens, top-p does not select a fixed number of tokens. Instead, this method sets a probability threshold.

Continuing the previous example, if top-p is set to 0.15, it means selecting the top 15% of tokens by cumulative probability as candidates. As shown in the figure below, the probabilities of "United" and "Netherlands" sum up to 15%, so these two become the candidate words. Finally, from these candidates, the word "united" is selected based on the probability score.

Top-p is usually set to a high value (like 0.75) with the purpose of limiting the long tail of low-probability tokens that may be sampled. We can use both top-k and top-p together. If both k and p are enabled, p acts after k.

Commonly encountered default top-p values are around 0.7/0.8. The same principle applies: if set too low, the model's output becomes too deterministic; if set too high, the model might become overly unconstrained, which isn't ideal either.

References

• https://docs.cohere.com/docs/controlling-generation-with-top-k-top-p
• https://docs.cohere.com/docs/temperature
• https://mp.weixin.qq.com/s/IswrgDEn94vy5dCO51I1sw