Finetuning LoRAs on BILLIONS of tokens 🤖

I just wanted to dive in a little bit more on your experience with the fine tuning team John Allard was publicly sort of very commentary supportive and you know was was part of it like what is it like working with them you've also trained on billions of tokens instead of the usual Millions range just like take us through that fine-tuning journey and any advice that you may have it's been so cool and this will be public by the time this goes out like open ey themselves have said we are pushing the boundaries of what is possible with fine tuning like one of the things we've noticed and it's a very interesting problem for them as well when you're building like a self- servify tuning API they have to decide how big your PFT adapter your low adapter is going to be in some way and like figuring that out is actually a really interesting problem because if you make it too big and because they support data sets that are so small you can put like 20 examples through or something like that like if you had a really sparse large adapter you're not going to get any signal in that at all so they have to dynamically size these things