Training & Fine-Tuning LLMs: Techniques

Key Takeaways

everyone welcome to the fourth and the final session of our training and fine-tuning llms course before we get started with today's topic please take a moment to introduce yourself in the chat share where you're joining us from my name is I'm a machine learning engineer at wasting biases connecting with you from Warsaw in Poland and we are absolutely delighted to have a special guest today for the second time Jonathan Franco Chief scientist at mosaicamal who will be sharing his expertise on trading and fine-tuning llm techniques Jonathan we appreciate your presence with us today thank you so much for having me I'm really excited to be back awesome so should I just go ahead and dive in yeah sure go ahead peace awesome let me share my screen and let's get started um so I'm Jonathan I am chief scientist at mosaicamel and now you know Chief scientist of neural networks at databricks and let's see hopefully you can see my presentation give me a thumbs up um okay looking good um so let's Dive In so game plan for today I really want to do three things um well four things I'm going to start and end with some friendly advice I give the same friendly advice last week um so for those if you're watching the presentation you know straight through um from last week you're going to see the same thing again at the beginning and the end um but you know I always like to give the same friendly advice whenever I give a talk so you get to hear it twice and then we're going to talk about Hardware requirements and cost we're gonna actually do some math here I do want to dig into the numbers and kind of get a sense for you know how much it actually costs and how long it actually takes to do some of this training and then think a little bit about memory and distributed systems and how to actually load these models and then you know when all said and done we can actually get in the finer points of pre-training and fine tuning um I'll actually start by saying one thing which is I hate the word fine tuning I absolutely despise it because I think it's a very poor description of what we do I've learned to live with it but whenever I say fine-tuning I think in my head the thing you do after pre-training that thing could be very cheap like you know training on a few tokens or very very expensive like taking a model like llama two and training it on another trillion tokens and so fine tuning is often not very fine and so I hate that word but I will think of training as divided into two phrases you do a self-supervised kind of more generic pre-training phase to create a good model for doing a lot of different things and then you may do many different things to it which we refer to as fine tuning even if they're not very fine at the end of the day foreign so moving along again my friendly advice number one always start small and work your way up and you know I've gone over this in depth already so I'll just go through it very quickly number two always be skeptical of what you read in the literature and test things for yourself number three don't trust your intuition again Tess thanks for yourself and I just want to emphasize this is science you have to run experiments and find out what's going to happen okay so let's talk about cost so here's how much it costs to train a network in you know a nutshell at a very high level so a flop or a floating Point operation is you know one computation on you know on a GPU or a TPU or any kind of ml accelerator usually a multiply ad or you know something along those lines and you know we we approximate by saying that basically all you know floating Point operations cost about the same thing and so we you know we look at a GPU and we say ah that GPU has a hundred teraflops and so it can do that would be a hundred times ten to the 12th um or a hundred a hundred times you know yeah times 10 to the 12th operations per um per second in that case floating Point operations per second so you know that's a lot of operations but we also need to do a lot of operations when we train a network so n here is the number of parameters in the network and so you know we have you know the number of parameters in the network times the amount of data you can think of this is just kind of the number of steps you take during training or kind of the number of forward and backward passes you have to do in the network then we've got this number six in here um and the idea is that essentially you know you've got about two operations per parameter on the forward pass and the backward pass is typically twice as expensive as the forward pass because you have to both update the weights and do the chain rule for the next layer back and back propagation I won't dwell on that too much but in approximation we often make with Transformers is you know if you think of it as a fully connected Network the number of flops is approximately six times the number of parameters times the the amount of data now I said if you think of it as a fully connected Network I'm ignoring self-attention I'll come back to why we can ignore self-attention a little bit later but let's just assume this is a basic approximation now the chinchilla scaling laws that I talked about last time tell you that you know essentially the amount of data that you should have is about 20 times the number of parameters in the network more or less so from this you can derive a pretty easy formula just plug in 20 times n for D on that first line and you get the number of flops that you know it should take to train a Transformer so I'm actually going to go through an exercise here I want to calculate how long it's going to take to train a 7 billion parameter model to you know the chinchilla amount of data on 64 a100s and I'm going to tell you that a100 runs at about 312 teraflops per second at teraflop is you know 10 to the 12th and in h100 is about 989 teraflops according to Nvidia okay so I'm about to show a bunch of math um first chinchilla optimal that puts us at 140 billion tokens that's just 20 times the number of parameters so here's the math I'm going to go line by line here as I just threw a bunch of numbers up there so let's take it one line at a time that first line I'm just calculating the amount of data 20 times 7 times 10 to the 9 that's 7 billion and so that gives us 140 billion so we can plug that into our flops formula and we get that our flops is 6 times the number of parameters 7 billion times 20 times 7 billion the amount of data and that gives us that number on the third line 5.88 times 10 to the 21. that is a really big number okay so now we can calculate you know the number of floating Point operations that we can do on this cluster per second ignoring Network costs and any other stuff that might slow us down so that's 312 teraflops so 312 times 10 to the 12th times 64 so that gives us about 2 times 10 to the 16th um you know flops we can process per second and then we just divide these two take my number of flops divide it by the flops I can do per second and that gives me the number of seconds and so you can see that gives me you know 2.94 times 10 to the fifth seconds that doesn't mean very much to me as a human being but 3.4 days does and that's how long this is going to take to train that's it awesome right I mean that's all you got to do to calculate costs okay so there's a little more to this um this was assuming that you were fully utilizing the a100 you were getting all 312 teraflops per second out of the a100 that's never going to happen in practice there are always going to be reasons that you're not fully utilizing the the a100s and so you know we have to take into account the fact that we're not you know we're not always going to get the most out of the hardware and so I'm going to add one more number here which is this mfu number mfu is it stands for model flop utilization and the way to think about it is you know how many flops of those available in the chip are you actually using properly and so the way you can calculate this is you can actually just run the model we can count the number of flops in that first operation and then we can just run the model for a while and see how many tokens we're getting through per second and then from there you can extrapolate well if an a100 is 312 teraflops and I'm getting this many tokens per second that many tokens per second every token I'm doing the number of flops on the first line and you can divide and figure out what fraction of the you know Max theoretical flops are you actually getting and that tells you the utilization of the chip that gives you that mfu number now why not just call it U I want to just call it utilization or Flop utilization y m model flop utilization so there are a bunch of ways of measuring utilization if you've ever typed in Nvidia SMI into your terminal you'll see you know to some extent kind of a momentary look at what percentage of the chip that you're using and often it's close to 100 or you know you're seeing some very big number um so it's important to keep in mind that not every operation the chip is doing is actually productive for moving the model forward I'll give you one example we're going to talk about memory in a moment but one thing that can happen is the activations that you compute for the model get so big that you actually run out of memory because you don't have room to store the activations one thing that people do is they do a procedure called activation checkpointing they will basically compute the activations and then delete the activations for some layers that saves you some memory and then on the backward pass you can recompute those activations when you have to using the other ones that you saved this saves you some memory but it costs you some compute because for some layers you have to do the forward pass twice that means that some of your flops aren't actually being used productively they're just being used for recomputation mfu doesn't take that into account because all you're looking at is the actual flops the model should have taken that first line six times n you're not looking at you know every single operation that was run on the hard work because many of them are redundant there's another you know another computation you can do for Hardware flop utilization or hfu which is in some sense how many actual operations are you doing in the hardware versus how many could it do we don't like to use that number at mosaic and you know I personally don't like it because it doesn't tell you the story of like how productive is training anyway long-winded way of talking about mfus but mfu is really really important if that tells us the actual utilization no you know if I want to go to this link I'm actually going to go to this link but this is our Mosaic ml throughput table this tells you you know what kind of mfu and hfu you should expect to get for a given model on a given Hardware at a given training sequence length this is for you know our repository it you know it wouldn't apply to any other code base but it should give you a sense a typical really good number is around 50 if you're getting 50 you're actually doing really well um so you know if we go and look this up the mfu for a 7 billion parameter model on 64 a 140 gigabytes for a standard sequence length is about 50.7 mfu we can then go back and plug that in on that bottom line here and you can see well I'm only actually utilizing 50.7 percent of the hardware so my 3.4 days turns into 6.7 days you can see why mfu is so important and a lot of people at a lot of places who are building llms sweat the details of getting another percentage Point here and another percentage Point here because it dramatically changes how long it takes if you could get this to 100 you'd have the training time that would be fantastic so that's what I wanted to say about training cost um before I go any further I do want to pause and see if there are any questions so Dirk you can tell me if there are any questions that have come in yeah um that I had a couple of questions um I can answer one of them there was a question about uh showing weights and biases visualizations we will not be getting into those details today but we can definitely take it as an offline follow-up and that is a question if we're gonna cover the difference between fine tuning versus instruction fine tuning I think that might be yes we'll talk about that a little bit later um yeah that is a question about um is it better to find you in a smaller llm for a special use case or to use like a like a bigger um bigger llm more General other than um I guess not sure if this is if necessary the right the right question for this part of the presentation but if you have any I don't know if it's the right part for this presentation but I'll answer it based on my friendly advice which is try both that's really the honest answer it's hard to know okay I think we can we can proceed awesome so there's one other thing I want to talk about which is memory usage um so you can start to do some back of the envelope math for memory so each parameter is typically represented in you know 16 bits or two bytes um so two bytes per parameter for the weights um Optimizer State often stores up to two full copies of the model depending on the optimizer you're using if you need first and second moments or things like that okay that's another four bytes per parameter depending on kind of how large of Optimizer State you're saving you'll see two bytes per parameter for the gradients and so we're already at you know eight bytes per parameter you can do the math eight times seven E9 that's you know 56 or 5.6 times 10 to the 10th if you divide that by 10 24 to the third remember when we're converting to bytes we do need to use powers of two so it's not you know a thousand bytes per kilobyte it's 10 24 bytes per kilobyte then we get to 52 gigabytes and you're going to run out of memory on an a140 gig okay that's not good and keep in mind that doesn't even include the activations um so we're already in pretty deep trouble before we've even gotten started okay so what do we do um you have to distribute you have to share across more than one chip there are lots and lots of ways to do this so the typical way that we might train a multiple gpus is data parallel and I'm stealing some great figures from colossal AI they have a nice guide on this so thank you to our friends at Colossal AI for the figures um and data parallelism is where you have multiple gpus you have a separate copy of the model in each GPU and then you just do different data on each GPU so then you can average the gradients when you're done this is not going to work for us because I just said we're going to run out of memory on each GPU we can't even have one model copy per GPU so data parallel nope out the door the next thing you can do is something called tensor parallelism where as you can see this is a nice figure from each Alaskan so on the left side here is a matrix multiply that we're doing multiplying Matrix a times Matrix B but you can actually split it along the columns and split it split b along the rows and in doing so you can create four smaller Matrix multiplies this is a way to save memory you can actually do you know A1 times B1 on one GPU and you know A1 times B2 on another GPU or maybe both in the same GPU but do A2 separately and this reduces the amount of memory you need for both you know the actual the actual weights that you're working on and the activations that you create so this is great um there's a much more complicated strategy called pipeline parallelism this is where you actually have each layer of the network on a different GPU and you're constantly sending data forward through the gpus and backwards through the gpus kind of conducting a forward and backward pass on each GPU depending on where the data is in the network and you it forms a pipeline you're constantly passing data forward to the next GPU and back to the previous GPU all the way through as you compute a batch um pipeline parallel tends to you know it's probably the most scalable of these approaches once you take everything else into account until your layers are too big to even run on a single GPU and then maybe you make them tensor parallel on a node the biggest challenge of pipeline parallelism is actually that you get these bubbles so what happens is when you send through that first batch all the other gpus are idle when that first batch is in layer zero and when the last batch is coming through the gpus become idle as you pass through that last batch so you need to do this on a lot of data before it makes sense because those bubbles do cost you a lot of utilization um now I've mentioned all this stuff but actually I want to mention I said data parallel doesn't work um I actually want to spend some time saying data parallel is great so I just told you this doesn't work because you know as I showed you before like I'll go back to that figure I just said the model is not going to fit on any GPU so what am I even talking about here so there's this fancy strategy called xero um and it's actually a set of strategies but here's the idea you know I have all this Optimizer state for atom but I don't actually need it until the very end and so while I'm doing my my passes on the network I can actually just take that state and I can Shard it I can break it up into let's say I'm training on 64 gpus break it up into 64 pieces and keep one piece on each GPU there I just reduced my Optimizer State per GPU by 64x and if I have more gpus it goes down even further now there's a cost to this that you know when I want to update the optimizer I also have to update it in a distributed way and when I want to use it I have to use it in a distributed way but you know sharding that Optimizer that just saved me a bunch of memory and that's stage one what's written here then you go to stage two you're getting all these gradients and so you know you're only a part of the optimizer so you only need the part of the gradient that corresponds to that so you can actually share the gradients and kind of split them up and Shard them among all the gpus that's awesome that just saved you a ton of memory keep in mind gradients were you know four out of our eight bytes and then you know the third stage is that even the model parameters are partitioned around the network so the idea here this gets a little tricky but the idea here is that if I'm using a layer of the network every GPU has to have that layer otherwise it can't compute the forward pass but once I'm done with that layer I can Shard it you know I can divide it into 64 pieces and keep one piece on each GPU and then I need to collect all the pieces the next layer on every GPU use that layer and then I can chart it again so I can keep them I can spread the model weights over the entire um over the entire network over the entire um over the over all the gpus and in doing so I can almost treat my gpus like one big shared memory instead of you know all these different individual memories so this process called xero is really really helpful the folks at Deep speed pioneered a really great implementation of it um today we tend to use fsdp or fully sharded data parallel this is built into pytorch you can just use it right off the bat um I I tried to find a snazzy logo but the folks haven't created one for fsdp yet so if any of the fsdp creators are listening right now um you could definitely use a snazzy logo um but we love fsdp at mosaic and you know it's it's integrated very deeply into our llm Foundry this is our kind of default way of training networks we'll use fsdp at scale and that's allowed us to scale up to you know hundreds of billions of parameters and you know thousands of gpus so and you know hopefully beyond that so that's really you know when it comes to distribution I think that's pretty much everything you need to know about training time and flops and memory and cost and utilization and then how to actually distribute it I want to pause again to see if there are any questions or comments that people want to throw in yeah we have we have a couple of questions um there's actually a request from some of the audience uh about the the quality of the audio so if if you can check Jonathan if there's any applications interesting close uh I think that would be everything should be closed okay um for those that have a hard time following uh the recording that we will post afterwards should have like all of the audio issues fixed so at least if you if you're better with us for some time like we'll we'll make sure to upload the high quality video but uh hopefully it works at least for some for some of the audience um I will post another questions in the question in the chat as well I'm gonna read it um the chinchilla lost estimation don't take into account many hybrid patterns and choice of architecture for the model those also not the inductive biases Allowed by the training setting do you think it's an overly gross estimate of the loss so it's a you know the the Crux the question is like all this chinchilla stuff it's a rough estimate um it doesn't take into account a lot of stuff and it may vary in every setting so can we trust it and the answer is well no we can't trust it um but you know I challenge whomever asked the question can you do better do you have enough gpus to do better do you have enough gpus to you know go and do this for a wide swath of hyper parameters or do it for a different model architecture so in some sense it's not perfect but it is a really helpful approximation that gives us some guidance on what to do and it's the best we're going to get to be honest um and if you wanted to redo this in every setting you'd be training so much that you know you'd have to train the network you wanted to train and 100 others or hundreds or thousands of others in the process and so it's just not feasible in practice so yeah you know and what's the famous quote all models are wrong some models are useful um chinchill is a very useful model that is completely wrong it's a great question appreciate it um maybe one more question from from me um we see all of your example actually based on on a100s and it seems like everyone is trading these models on a100s um how about some of the older gpus that might be more available like v100s or D4 maybe like um does it make sense to consider them for for training or fine-tuning llms or do they have some deficiencies and we should default to a100s it's a great question so v100s are excellent gpus they're thoroughly outmoded at this point compared to the latest stuff if I recall correctly a V100 will get you I'm trying to remember the exact number of teraflops for a V100 but um you know nothing that will blow your mind today but what blew my mind back in 2017. I'm now actually Googling it a V100 has 130 teraflops um so it is about you know a100 is about two and a half times more and an h100 is what is that about six times more uh seven times more so you're in a place where you know in some sense if you told me I could have any GPU I wanted um and you told me the prices and you know based on the prices I tend to see on marketplaces I would take you know I would look at those prices and say wow V100 is not you know not really that competitive because modern gpus are just so fast but you know if you have nothing else a V100 is still an excellent GPU and you can get a ton of work done so you know definitely not a bad thing the biggest trickiness with v100s is actually if you wanted to do a multi-node training job if you want to go more than eight gpus if you wanted to do that you'll be in a very tricky place because many of the v100s were not built with very high bandwidth between them it just wasn't the days of doing this large-scale multi-node training in the clouds and you know nobody's upgraded that because they've moved on to new things so you'll end up in a place where things may get a little slow if you try to do multi-node training but at the same time still an excellent GPU when you get to things even older than that things start to get very tricky the V100 was the first mainstream Nvidia GPU that had tensor cores and the tensor cores are built for things to run really really fast if you go back to say a p100 you'll no longer have those and things get tricky thank you so many other questions we should cover should we should we keep going I think we should keep going thank you awesome so now we get into the question of architectures what should you train um and I'm going to give you more questions and answers in the spirit of test things yourself and don't believe what you read first architect exercises and scaling are pretty standard um like you know we we have standard ways to increase the number of layers or increase you know the width of the network as we scale up you can kind of follow the best guidance from you know the chinchilla paper or the opt paper or if the Llama models did there are a lot of excellent references where those folks did their homework and you know we can trust largely what they found at least you know not very many of us have the number of gpus necessary to go and question what they found and try to do better but there is a trade-off between training and inference costs and this is where chinchilla kind of comes back you have to take into account well how fast you want inference to be and how much work is happening at inference time you could train a chinchilla optimal model but that model may not be optimal for inference depending on you know if you've got a lot of inference queries you actually want a smaller model it'll do inference more efficiently it'll be lower latency it'll be cheaper to run and in that case it might actually be worth your time to train a sub-optimal model at training time to train longer on a smaller model this is exactly what the Llama folks did that was kind of the Revolution to me of the paper they you know they train for a very long time on some small networks and still got pretty good results even if they were sub-optimal compared to what they would have done if they had spent the compute perfectly for training but the networks became really popular in my view because they were easy for anybody to use at home and do inference on you could do it on your gaming GPU that was a big deal and so really at the end of the day your architecture Choice involves this trade-off you need to make between you know how much you want to spend for training and are you willing to actually spend a little bit more for training to get cheaper in France and that's a trade-off you can then calculate um the next big decision you have to make a sequence length sequence length is tricky um longer sequences increase the cost of attention because you know the self-attention operation is quadratic if you have a longer sequence then the model needs to attend you know every token has to attend to every token and so you end up paying this very big penalty um and you know it gets quadratically worse as you increase the sequence length now it's important to note that with some Modern you know techniques like flash attention I won't talk about Flash attention but it's just a fast way to do attention um the attention costs are relatively small compared to the overall cost of using the feed forward layers in a Transformer Network and so you end up in a place where actually you'll see before I didn't even take attention into account when I was Computing the flops because attention is so negligible and so fast compared to all the other fully connected layers in the network so you know eventually you get long enough sequences that attention cost does actually start to matter and it's kind of interesting if you look at some of our throughput tables you'll see something pretty remarkable as you get to really big Networks there's basically no penalty for having longer sequences because the cost of attention is completely drowned out by the cost of the bigger Network for a 30 billion parameter Network I believe there's no penalty and going from sequence like 2048 to 4096 and only if I recall correctly correctly a 15 penalty and going to sequence length 8192. that's not too bad that's you know price potentially worth paying for a 7 billion parameter Network the the kind of relative price to you escalates pretty quickly because the rest of the network is so much smaller um then you get to a question of you know just data do you have enough data to use a longer sequence length it's a tricky question we found that in a lot of the standard data sets that are out there there just isn't a lot of long sequence data so if you wanted to train a long sequence model you may not have much to work with and even if you did have stuff to work with you have to be very careful where you pull it from if you try to kind of sample from all your data sets evenly some data sets don't have any long sequence data and you'll see the same few examples over and over and over again then we get into one last open question I think this is more of an open scientific question right now just because we can cram really long sequences into memory does not mean the model is actually taking advantage of them you know just because you can get the bits from point A to point B doesn't mean that you know you're able to you know actually use them in an effective way the model may not be able to learn or may not be able to holistically take advantage of all that information and I think that question is still out there we still have a lot to learn about long sequence training so there are so many other choices I'm not going to go through every single one I lumped all these choices together into one slide I'm going to go through them but I lump them together because I don't think they're clear answers to any of these positional encodings how you represent you know which token is in which part there are lots of different methods the two most popular rotary embeddings are rope rotary position embeddings and Alibi rotary position embeddings actually learn position embeddings Alibi just adds a bit of a bias to you know the attention operations of things that are further apart we use Alibi at mosaic because it's a little bit faster you don't need to really change anything about the network you just need to you know plug in a little bit of a bias into the attention operation there's no real problem with that rotary embeddings do slow things down a little slow things down a little bit rotary embeddings do tend to be a popular choice or rope whenever I say rotary weddings I mean rope here rotary positioning Coatings um you know the Llama models used rope um I think the folks at adeptide I mentioned that they used rope for the 8 billion parameter model they released a little while ago and so you know it's a choice that I don't think really is settled right now we used to say that alibi was really good at extrapolating Alibi automatically gives the model the ability to kind of look at longer sequences than it was trained on but now there are lots of interesting ways of also interpolating to get extrapolation with rope embedding so it's a little bit of a mixed picture there okay next up is the activation function um this is also a bit of a mixed picture you see people using all sorts of different things you see relu just kind of a standard rectified linear unit you see gated linear units which actually kind of have their own I think of it as almost an attention operation inside the activation then you have Swig glue which is gated linear units but use a switch activation there are a lot of different choices here and again you see many different choices in many different models right now we use relative Mosaic I believe that the Llama models used a different I believe they used Swig Lube but I may not have gotten that exactly right and you know there are a lot of different choices here a lot of different received wisdom but no real definitive answers as far as I've seen okay then you get to optimizers and atom is the typical default Optimizer that people tend to use but there are a lot of challenges with atom mainly that it just takes up a lot of memory you need two full copies of the model in terms of memory just because you have to you know you have to fit in the first and second moments as you're training and so there tend to be there have been a lot of other optimizers there are eight bit variants of atom that kind of reduce the memory by half there's an Optimizer called lion that came out this spring that also kind of has some of those behaviors another Optimizer called Sophia that came out that claims to be better there are a lot of different choices here and I don't think there's a definitive answer my recommendation when there aren't definitive answers is you know choose something that's popular look at what we did at mosaic look at what our friends at meta did um for the Llama models and you know those are reasonable choices that are pretty battle tested if it was able to produce a good model you know it's something hopefully you can believe in and stand by um so you know that means Roper Alibi llama used rope we used Alibi real user fine um you know Adam is fine there are a lot of different choices last part is the tokenizer so I'm not going to spend too much time on tokenization today because I could probably talk for years on tokenizers and I'm not an expert in that and the experts could probably talk for even longer tokenizers are you know how we translate actual text into individual discrete tokens of the model can process and there is a whole art and literature to this this is a very big meaty Topic in the NLP world that people have spent a very long time working on um there are lots of different tokenizers byte parent coding or bpe and sentence piece tend to be the two most popular from what I've seen and you can train your own tokenizer that is develop use a tokenizer to develop its own kind of vocabulary on your data and or you can use a generic off-the-shelf tokenizer like the GPT Neo X tokenizer developed by the folks at a Luther AI there are pros and cons to building your own tokenizer building your own means that you're taking the risk that maybe your data doesn't represent what people are going to use the model for and your model may not be as efficient or as effective at processing that kind of data there are benefits though if you're dealing with code for example the kinds of tokens you get with code are very different than the kinds of tokens you get from natural language text you'd get you know if and for and curly braces and things like that it also helps a lot if you're dealing with a language that is not well represented in existing models you know if you're dealing with a non-english language essentially the kinds of tokens that you're going to want are probably going to be very different or the vocabulary that you're going to want is going to be very different and it might help to train your own token either the ask that I make is if you're going to train your own tokenizer it is a messy difficult art and I strongly recommend comparing anything you create against a good general purpose tokenizer like the GPT Neo x one just to make sure that you actually did a better job um you know certainly it's a very difficult thing to do and not all the tokenizers we've created at mosaic are actually better than the generic one and so you need to evaluate your models it's also worth keeping in mind that if you change your vocabulary size and even if you change your tokenizer loss is not necessarily going to be comparable between two models so you do need to look at Downstream metrics I want to pause here again for any questions about you know architecture and design choices in building an llm this this has been super helpful I I was actually planning to ask about like what would be your default recommendations but I I like this like popularity um uh rule like just pick what what most uh most Frameworks are using picword has been proven to work I kind of think that you know unless you have a huge amount of GPU sitting idle and you can go and test this all for yourself the best thing to do is you know work for llama works for you worked for MPT works for you um you know you're not going to do worse than those models maybe there are better choices out there but llama's a pretty good model you can't go wrong if you follow what they did maybe one question because I I think for that for the vocab sizes um I I've seen like a really broad range from like uh 28 000 tokens to like 128 or even more do you have any any good heuristic on on how to choose the backup size it's a little tricky because the vocab size will impact the efficiency of the model and so you know you need a bigger vocabs will be less efficient but bigger vocabs mean you have more tokens and you may be more token efficient that is to say a piece of text may fit into fewer tokens and kind of get back some of that inefficiency there's a lot of debate open AI from what I understand tends to go with very large vocabularies other folks I've worked with like very small vocabularies um in some sense it is in the eye of the beholder and that's something that I don't think has really been definitively studied and it probably varies by domain quite a bit so the answer here is like a standard 50 000 or so tends to be pretty popular I don't think you can go wrong with that um but I have seen you know down to 25 000 and up to over a hundred thousand before in production and none of them have been like make or break in terms of making a model much better or making a model much less efficient I think somewhere in that range you can make a very safe choice thank you maybe one more question on optimize this I I've heard like some people considering SGD for also for training uh or fine-tuning at Adams because it doesn't have this memory overhead of Adam and do you have an opinion if sdd is a valid Choice as well it's complicated I've certainly tried SGD for pre-training and it has not gone well my experience as a computer vision researcher was always that kind of you use Adam when you don't have a better choice or you don't know of something that's more specific for the problem Envision they used to use atom until we had batch normalization which smoothed out training and reduced a lot of the instabilities and then everybody switched to SGD and SGD tends to outperform Adam in good in well-tuned settings and computer vision for Transformers I sometimes wonder is Adam the right thing or maybe we haven't gotten training to be well behaved enough that we can do something more specific so I think it's an open question when it comes to fine-tuning I don't really have any clear guidance there I think it is going to vary setting by setting it's hard to go wrong with Adam if you know you you really sat me down and said you get one try at this and you can only do one thing I would say okay I'm doing Adam but if you have multiple tries and you can play around a little bit optimizers can make a huge difference in terms of quality of results after all a trained neural network is just solving an optimization problem so you know different solution better solution they are out there so it really depends on your budget and how much risk you can take thank you I think we can go on this has been super helpful awesome so then you train the model um and all hell breaks loose um and you know I say this a little bit tongue-in-cheek but also especially in pre-training there are lots and lots of stories out there and a lot of experience that I've lived through where pre-training has just gone horribly wrong for reasons that people don't always understand um you often see lost spikes and you know I'm showing you a couple of examples here from internally at mosaic ml where you know suddenly unexpectedly loss jumps up and sometimes it comes back down and sometimes it catastrophically you can look at this model on the right you just kind of catastrophically kills the model it's a big problem I will also note experiment tracking I just want to give a shout out quickly because somebody asked about it um you know we do use weights and biases at mosaic ML and you know this is a weights and biases event these screenshots are from weights and biases where you know I went and found some models that had lost spikes and took some screenshots of them um so there are lots of different ways to address lost spikes there's some algorithmic ones one of the most popular ones you'll see in the literature is simply roll back to a checkpoint change the random seed maybe change the learning rate lower it um and then try again that's you know that tends to be kind of the state of the art in some sense from what I hear people describe in a lot of practical settings there are other proposed approaches and a lot of research going into this topic but I do think it's kind of you know it's it is a tricky thing and it does bedevil especially you know I think in the single digit billions and above you are going to run into lost spikes if you're trying to push your learning rate high enough to get state of the art results um next thing I'll mention is Hardware failures um you know a lot of gpus die um this this may sound obvious to some people and it may sound surprising to other people but gpus die pretty frequently or gpus run into different issues I have great stories about all the creative ways that gpus can die um and I think I lost a slide here but you know what I what I was going to talk about here is that they're really you know a bunch of challenges when you lose a GPU um the first is that once you lose a GPU the training job dies um we haven't built fault tolerant ways to train and so your training job dies and you know what do you do um you just have to resume from a checkpoint and you know pick up some more gpus that are alive and keep going um there are a lot of challenges here and even detecting a dead GPU I've seen situations if the GPU isn't dead dead the GPU is straggling and running slowly and you have to detect that and then you know get rid of that GPU and keep going the GPU is also die in groups of eight because on all the major Cloud providers they're soldered to the motherboard A2 motherboard this is how you get the fast you know multi-gpu kind of you know multi-node interconnect that Nvidia provides or that any provider you know offers you and so you know you you can't really just swap out one you have to swap out a whole board or leave out a whole node the you know the next big challenge here is that you know you got to pick the the size of of how many gpus you're dealing with very carefully because let's say I'm running a job on 512 gpus and one GPU dies my job may not fit anymore if I split my checkpoint to 512 pieces how do I even resume if I picked my data set around you know having 512 separate workers on it what do I do when I lose a GPU I've you know I may lose determinism if I try to resume on 511 gpus or 504 gpus if I lose a whole node it's really nasty because as your training jobs get bigger and bigger it's more likely you're going to encounter a GPU failure there are just more opportunities for that to happen but there's still no fault tolerance so it's your most expensive highest risk jobs that are the ones that are most likely to confront a GPU failure just by probabilities and that's really really challenging so what do you do I have a few pieces of advice here because this is a persistent problem for us and in general um first is there's a lot of work you can do to automatically detect failures you can look for NVIDIA errors you can look for kind of the failure conditions like the job suddenly gets really slow and identify those proactively because a lot of the time your job won't just crash it will just get really slow or get stuck somewhere and oftentimes the error messages can be pretty cryptic if you just look at what prints out on the command line and aren't monitoring the gpus it also tends to be helpful to keep spare gpus available um you know I generally never train with a full cluster I'll always keep a few gpus on the side so that I can swap them in as necessary and you know you can you can still use those gpus for low priority work you can still get things done on them but you know they'll just get taken over if they're needed you can use sharded checkpointing so one thing I forgot to mention before is that the other problem with resumption your checkpoints are really big we can do some math here but just taking you know Optimizer State plus GPU or plus model um that 6X the number of parameters in terms of bytes so if I have 100 billion parameter model that's 600 gigabytes that is huge in fact in many cases that may be too big to fit into memory on a single um on a single node depending on how big your memory is but it's also very tricky because it takes a long time to save that to storage and to load it back from Storage pytorch has introduced some really great support for sharded checkpointing and we've taken advantage of that in our Mosaic work so that you can split the checkpoint up into if you've got n gpus you can split into n pieces and so that way each GPU can use this network bandwidth and storage connections and save much more quickly and load much more quickly that tends to help a lot um and the last piece here is that you know there are data loaders that provide Random Access again if you load from a checkpoint you may end up having to spin all the way through your data loader to go back to where you were and you know we have a data loader that I mentioned last time at mosaic that allows you to just have random access and our data letter actually will do deterministic training even if the number of gpus changes and we have you know what we call elastic sharded checkpointing where you can actually have a checkpoint charted for let's say 512 gpus and resume it just find on 504 gpus or whatever other number you want so there are a lot of mitigations some of which are built in the software some which involve monitoring Hardware some of which are just best practices but it is something you need to be very careful about um and then you fine tune so we get to the fun part we fine tune I'm going to say just a little bit on this so someone asked about fine tuning versus instruction fine-tuning or lhf is in there and I want to I kind of left the slide blank to answer any questions that we had there so I'll clarify a little bit and then put out a call for questions um but the way to think about fine-tuning is that you're taking a model and you're training it and using it for some other purpose that purpose may be more specific it may be a particular domain like fine-tuning a model for code maybe a particular task like classification it may be some other paradigm and you know what tends to be popular for general purpose chat Bots is a process where we start by fine-tuning for anything fine-tuning is kind of the umbrella term and you know one thing we might do is what's called instruction fine-tuning or supervised fine-tuning sft or ift sft is becoming more popular so I've started switching to that and the idea is that you have a lot of example interactions with the model a command or an instruction and then a response and all you're doing is you're kind of training the model to you know see its first part of its input as an instruction and then respond to it because naively the model is just trained to spit out documents it's trying to just kind of continue generating tokens and giving it some examples of instructions and responses is really helpful and I mentioned last time you know on the order of tens of thousands we'll you know get you into the ballpark of llama 2 in terms of what they did as long as they're really good quality and quality matters then there's a process called rlhf and I'm gonna you know say a little more about our lhf hugging face has this beautiful blog post on how our lhf works and I recommend reading it but you know I stole a couple figures from it and the idea is that you know we can ignore the RL part I think almost you know there is are all going on here but I think it over complicates things in terms of understanding all you do is you take your language model you generate some outputs with respect to some prompts and ask humans what they think of them was this good or bad was this one better than that one how much better was this one than that one and then you train a model to predict a score for each response like how good was this response or how good would humans say this response was and then you go through this more complicated process as the RL part so I'm not going to dwell too much on this but all you do is you essentially train your model to you know in to maximize the reward to give props that humans would give high scores to based on this reward model you can also you'll see this initial language model over here you can also kind of you don't want the model to stray too far kind of get to um you know get too obedient in some sense or kind of you know try to max out the reward so you can also constrain the model to behave like the initial language Model A little bit so that it doesn't change too much so you know that's the rlhf process and the idea with rlhf is that you you know you're trying to get the model to align better with human preferences when people talk about alignment this is often the technique that they're looking to use okay the last thing I'll mention is Laura because Laura tends to always come up in the context of fine tuning and I didn't want to go without mentioning it so when you fine tune often what happens is you're modif