LLM Knows The Future (by Apple)

Key Takeaways

Hello community. So great that you are back. Yes, today finally we go beyond the next token prediction of our large language model and we will have an AI that can look into the future. Unbelievable because we have a new AI research and that tells us hey you know what your LLM already knows the future of your tokens. And here we have a brand new team by Apple and they tell us your LLM knows the future uncovering its multi-token prediction potential of UI and this was published July 16, 2025 and finally finally we go from next token prediction to multi-token prediction. Now, Apple makes here a fascinating scientific claim that the standard auto reagently slow due to the token by token generation already possesses here a latent untapped knowledge of the future tokens well beyond just the next token. And the orers now propose here a novel framework for AI. But the beauty is we don't have to pre-train a new model. We can use our existing LLM and there's a new methodology to unlock this hidden potential in existing pre-trained LLM. And Apple tells us, hey, we developed a new minimal finetuning which will enable a model to predict the whole chunk of text in a single forward pass, dramatically accelerating your inference speed. And Apple tells us the speed ups without sacrificing you the quality of the answer by the eye will reach 500% for code and mathematics and 250% for your chat. Now imagine you don't have to buy a more powerful laptop or a more powerful PC 500% just by a little bit fine-tuning an existing pre-trained model. Let's give this a try. Let's try to understand what happened. And I know a lot of you might say, okay, hey, but it's Apple, you know, look at Siri. I know my Siri, I mean, at least here in Europe, has not developed any additional intellectual capacity in the last five or 10 years. Nothing. Zero. So Apple, should we trust Apple? But you know what? This could dramatically reduce our operational cost for our application. It would enable real time a high throughput 500% and it would have here a real responsive AI assistant even on smaller gadgets. So let's say okay we we know that it is Apple you know but imagine all those thousand of science of scientific researcher sitting there in the donut in Silicon Valley not allowed to develop their own eye. So they have some ideas so why not give them a chance. So let's start now. The beauty it is not about retraining like I showed you in my last video. A multi-billion parameter LLM from scratch. No, Apple thought about a very elegant fine-tuning recipe that preserves the original model powerful capabilities while teaching it a new faster way to speak to generate the next tokens. And this is really a masterclass in augmenting a system while respecting its original design let's say by openi or by gro or whatever you like. So let's have a look at the future of AI and Apple did a simple experiment. Honestly have a look at this. They said hey we have a prompt. What is 2 + 2? I mean, it can't be easy, you know. And they just append here several placeholder tokens to the prompt and examine the model's prediction here at the first step for those placeholder tokens. What would be here the output in those tokens? And when looking here at the output logic here for the first token, the hidden representation in the neural network, the word two was highly ranked of course and for the second one plus was ranked high and so on. So the entire correct sequence here 2 + 2 = 4 was indeed found within the top 200 logit prediction for this future positions within the hidden layers. Now you say okay so I found it but it was within the top 200. Absolutely. And now the idea is bring it here from the top 200 to the top one position. Turns out rather easy. Let's have a look. So the central revelation is the LLM in a single forward pause computes the hidden state that contain rich information about a probable futuristic sequence, not just a single next token. Well, how beautiful can this be? Well, we need exactly five additional ideas. And Apple showed us what we need. We need a mask information formulation. We g we need here a gated l a low rank adaptation. We need a specific sampler. I will show you this is simply on MLP. We need a quadratic speculative decoding algorithm and a new loss function which Apple calls a latent consistency loss that is absolutely easy and simple to understand. Where do you think we start? So let's have a look at the future. The future is in the logits of our LLM. And you remember we have we have to train this. No, we do have a model. Hopefully, it is open source. So, we have access here to all the weights and everything. And now we have to do just an add-on training of fine-tuning with tokens. And to enable here multi-token generation with a minimal retraining, they introduced here, yes, you know, there's no special tokens called masks. I showed you this already five years ago with the bird and sentence transformer and we have a sequence of token and the idea is to bend k unique mask tokens to the end of the sequence and form an augmented sequence here now with the mask token and the representation of the mask tokens are generated as random vectors and added to the embedding table and we refer here to the standard next token prediction as NTP next token prediction and a prediction for the mask token is MTP grid. So what we do we have here an input and now we want to here add two extra tokens and you see here the mask token M1 and M2 is now added here in a visualization you see here to every input. So we have the NTP and the MTP tokens when they are now in blue and the mask tokens are now in yellow. This is easy simple. Now of course we have to do the training. So what do we do? Now if you remember here now first let's start here with the augmented sequence. The augmented sequence is now fed into our decoder. No the network produces here latent representation Z1 to ZN. And here you have it. Yeah. Beautiful. And those are produced here for the blue tokens for the NTP tokens. And then of course since we have here the mask tokens we also have here set n +1 to set n + k for the mtp tokens. And this is here in yellow. So standard procedures as always. We just added here some tokens and we produce here some latent representation for two classes of tokens. Nothing special about this. Now comes a nice idea. And now we add a sampler head to this. Why? The first NTP token is generated classical auto reggressively using here the standard unmbeding layer. Great. But the remaining MTP tokens now there is a little bit more complexity for producing them. They are now produced sequentially by a sampler model. And we have here added complexity and relation to already produced NTP tokens. So instead of just predicting here a token here the y n +1 only here from its corresponding hidden state the set n +1 the sampler now conditions now the prediction on the previous sample token y n +1 as well. So the sampler does not just generate here the new mask token but the sampler takes into consideration the already produced NTP token in a linear sequence. Of course you have a linear dependence here in the sequence. Now the sampler model what it does it creates here a fast recurrent dependency chain within the multi-token generation step ensuring that the output sequence is here coherent in its linear sequence how we build it in the easiest case Apple shows us here a two layer MLP block multi-layer perceptron block to serve here as the sampler module this is a triable this is a simple module we know for years nothing special here. But now, now comes another idea by Apple and they say, "Hey, wait a minute. We know Laura." Now, 2 years ago, I produced a dozen videos on Laura here on my channel. And now they say, "We have Laura, but now Laura, we decouple now here the NTP from the MTP tokens." And only the MTP tokens get here a gated Laura mechanism. low rank adaptation matrix multiplication. This makes absolute sense no to enable now the finetuning we want to preserve the pre-trained model behavior. So this means we want to leave here the performance of the NTP tokens untouched because maybe it's an excellent model already. So we augment only the decoder layers here with the gated Laura modules and during the fine-tuning exercise only those Laura parameters and sampler head parameters are updated while the original decoder weights remain frozen. This makes sense. We just want to extract here the future token and the next token prediction is great. We don't touch it. So this gated Laura module now ensures here that the finetuning does not affect the NTP token behavior by applying here distinct functional paths for NTP and MTP token classical Laura nothing special about it but implement it in a clever way. So you know because normally with fine-tuning what we have as a classical risk factor is catastrophic forgetting. we override knowledge. No, now we don't want that with an MTP token with a mask multi-token we override here the single next token prediction. So therefore we have to find a mechanism to counteract the catastrophic forgetting and the gated lower here on the MTP path only solves this problem because it guarantees that a model standard autoraphic performance does not degrade which is a massive advantage over other fine-tuning methods. So what a clever way by Apple to have here only a selective path here for a gated lower application. Now let's come to the prediction of our multi-tokens. Now, now Apples tells us that LLM can now propose a chunk of K K 1 2 3 4 maybe eight future tokens in one run. But are they correct? That is, are they the same tokens the original unmodified LLM would have generated in our classical one by one token prediction? To make sure that this condition holds is valid, Apple introduced here a speculative decoding optimization. Nice. Let's have a look at this. So what can happen if you just do it naively? No generate k + one token in each generation step. The naive method can now lead to incorrect outputs. So we have to correct this. Especially if we have longer sequences of K longer than four, five, six, seven tokens, we get incorrect outputs. So how we do this? We simply have to verify whether the predicted future tokens are valid. So especially if the K+1 tokens generated here in a single step match what the model would have produced over the standard auto reggressive step the next token prediction if you run K + one easy idea so let's do this so we do this by a speculative decoding system now the order thought about it and said hey we have more or less two ways to do this the first here is a linear decoding stop here you have step t and then step t + one and now we just do a verification of the tokens. No problem. Um if we have a linear chain of those if for example one verified token now fails then the complete remaining chain of the remaining speculative token is not valid or is discarded. Therefore this is something we say okay this is one idea but it's not really helpful. I want if a validation fails at any point that not the complete sequence is lost. It is great but this is not working for us. So Apple came up with a quadratic decoding but careful the quadratic is not in the term that you think it is. This is just a more robust approach. So it is more or less here a new methodology that kind of guarantees here K for K equal 8 new speculative tokens are generated at here at every step and the key is now to interle the mask tokens with the speculative tokens from the previous step. So we will make baby step that have an internal reference. Here you have here the simple formal the input for the next generation step at t +1 is now constructed in this sequence. So you see you always have interlift mask token what it brings us this approach is is something let's have a look. So even if the verification now fails at some point the model still processes here the subsequent mosques and generate a new set of k specific token from that point onwards. So even if we have a single fail in a token prediction, it's not lost. The following sequence is not gone. We still have in the process the generation of a new set of K speculative tokens. And this makes it a complete decoding process so much more robust and ensures that we have a high consistency and acceptance rate. And the name quadratic decoding is here simple from a mask in the expanded input. Now Apple said you know what they said we can further improve the performance of the system. We can boost the performance of the system if we introduce a new loss function. And they called it here a latent consistency matching loss LCM. The most important term here is of course the matching loss because what we want to achieve, think about it, the goal is to encourage you the M's hidden state for a mask token prediction to be as close as possible to the hidden state for a standard next token prediction of the same token. We say we want to have almost identical token, but we just want to have it right now and we want more than one. So what we have now if you want it is just to be as close as possible. So guess what we just look at the difference that we can achieve. No. So very simple formula here we have with Z subt is here the anchor the hidden state from a standard auto reggressive stuff. This vector is here detached from the computational graph. So it doesn't receive the gradients. And our s as a function here of set of subt is the set of the hidden states from the mask prediction that should correspond to set subt has to state for the fourth mask in what is 2 + 2 and the mask structure. We just take here the difference and we do here an optimization. Simple. If you want you can think about this as an act of a distillation of a self distillation. because this is more or less teaching here the MTP path wave to perfectly mimic here the behavior of the original NTP pathway. Yes, of course, we want to have the difference as small or neglectable as possible leading now to a higher acceptance rates in our speculative decoding in our generation of those tokens. What a simple and elegant solution to find here this particular solution to our problem that we want to have our model hidden state for a mask token prediction to be as close as possible to the hidden state for a standard next token prediction. Great. Now let's talk about the result. Let's talk about performance. You're mask and I ask what is the transformer model that they trained? It was a llama and they had a supervised fine-tune version here with a particular data set and this data set was the Tuluo 3 data set covers a wide range of task including mathematic coding general conversation safety benchmark a strong candidate to test the general applicability of the method that Apple developed. You can find the 203 data set of course here open source on hugging phase you can download it everything is available for you. Now you might say okay so on and what parameters the training process happen what would Apple now recommend after running through all the optimization process now and they said they trained a model here with K equal 8 mask tokens. So system was designed to predict up to nine tokens at a time. So we have the one standard next token prediction token the standard token and eight MTP tokens mask token that go in one run for a multi-token prediction. This is the future. This is the view into the future. What are the next token that will come up here in the probability distribution and to preserve here the base model capacity the performance they simply froze the weights and only fine-tuned here two added components as I already told you the gated lura layers and they specifically ran with a rank of 128 I explained this in my videos where I explain Laura and then for the sampler for the hat that they added here as I told you this This is simply a two layer MLP. Yeah, hardware infrastructure. It was fine-tuned on eight Nvidia A100 GPUs, 50,000 generation ADMW optimizer. Classical way to do it. And here we have the results. So here you have now for all the different benchmarks here. What are the skill? This is about knowledge. This is about math. This is about coding. This is about chat. This is about safety. Great. And now let's go for k= 1, k= 2, k= 3 up to 8. And we have here the speed up contained by molten token prediction of a different domain for this tool 3 data set for a llama 3 model. So a very specific data set, a very specific model, but in general you can run any of your models that you have at home open source with this particular methodology. And you get here the speed up. You see here for coding you have here 5.35x times performance here in the speed up or if you go for knowledge 2 and 1/2 times a speed up for knowledge generation. But you know one of the most compelling findings I personally see here in the analysis here of Apple here is here something that is connected with the rank of Laura because you would assume that you have to go with a high ranked L'Oreal configuration. But look at this they analyze here the effect of the Laura rank here in the middle you have the rank of Laura and you see 1 2 4 5128 but look at this. Yeah. On the different colors here, one mask, five mask or eight mask. So you see, okay, the more if you want future token prediction you add, great. The higher the speed up, great. But let's have a look at the rank of Laura. You see we have almost a saturation setting in quite early. Then we have a little peak here at 128, but at 512 g it comes crashing down. So if we operate here in this let's say saturated region we don't need 32 as a rank of the lower calculation a rank of two might be absolutely fine. Now this helps a lot because if we look here at the memory overhead in percentage you might say okay hey just 1% 3% 5% yeah everything counts but look it if we have now let's say rank four rank four is below 1% of additional memory overhead this is fantastic this is a methodology to have more than the next token prediction or multi-token prediction with masks for a lower rank of Four is already is kind of helpful here. What a beautiful configuration. We don't need to go here to the higher ranks of Laura. Great. Low rank is enough. Memory overhead is practical neglectable. Yeah, adding the sampler hat to did a lot of ablation test provides you a much larger performance gain than increasing the Laura rank. We just saw here there is a almost a let's say a horizontal performance indicator and the last one here is simply if you increase the rank beyond the 128 I showed you it can hurt the performance you fall back and they think is they have such a large adapter on a relatively small fine-tuned data set this simply leads just to overfitting if you have no additional benefit from performance update and this shows us something you don't need a massive costly adaptation to unlock a potential that is already there in your open-source LLM. You can have a tiny targeted nudge. This is absolutely efficient. A rank of four with a Lauren model making this method remarkable efficient in terms of both the computation and the parameter overhead less than 1%. And you might say, okay, so I have a speed up. Great. It's working. Great. But are you sure about the quality? Because sometimes in happen if you increase speed for 500% you have to give something. No something has to give. So critical question is the speed up comes at the cost of the quality. Now think about our structure. We had a lure but we had not just a normal lure applied to every token but we had a gated structure with our lore implementation. So the classical quality cost you store is completely different now because our fine-tuning process you notice is now only on the MTP tokens and you can see this and Apple sort about and said hey there's an easy indicator for us and this is the cross entrance directly but now from the NTP from the next token prediction tokens if we look at this loss function and we and we look at the behavior of this loss we immediately see if it's working or not. So here you have another experimental result. You have on X the iterations here on on Y the loss and you see for the entropy of the entropy cross entropy for the standard Lura and this is the loss increases meaning the model gets worse at its original task if we go here with a standard Lura but we don't do this ABA said no we have chosen here particular gated Lura architecture only on the MTP token and you see perfectly flat no increase here in the cross entropy Of course, because we build it that way, the Laura puff is disabled for the NTP token. So there's no gradient flow back to them and the model's original performance is perfectly preserved. And here you have, if you want the experimental validation of this and of course we build it in this way, just to make sure that there's nothing happening here somewhere deep in the neural network. Now we can say now this method now achieves a significant speed up without any degradation in the quality of the text generated by the model. Absolutely amazing. Such a simple sort just five steps of optimization. Just taking elements that we already know building up a loss function that just looks at the difference. Come on Apple. Really? We had to rely on Apple that they could develop this amazing. So here we are AI the AI system of the future. What a great insight. We can use our classical LLMs and Apple has given us a simple supervised fine-tuning methodology. Just a tiny little notch that we have to add and our AI can have a speed of 500%. Absolutely interesting. I'm gonna try this out right now. See you in my next video.