Speculative Decoding
The standard way of decoding in the LLM architecture is to generate one token at a time. Speculative decoding is a technique to generate multiple tokens at once.
-
Models which run through the entire neural network to predict the next token are called dense models.
-
Models which run through a part of neural network to predict the next token are called sparse models. Sparse models are quick and have only a fraction of the entire model parameters active for a given request. Mixture of Experts (MoE) is a type of sparse model.
-
Methods such as DSpark, DFlash, EAGLE and Drafter use a secondary smaller model to predict the next token faster. The main LLM then has to decide and either accept or reject the predicted token.
-
MTP (Multi Token Prediction) is an LLM architecture with a separate MTP module. This module is trained to predict multiple future tokens.