Training & Fine-Tuning LLMs: Data

Key Takeaways

hey everyone my name is I'm a machine learning engineer at weights and biases joining you from Warsaw in Poland let us know in the chat where you're joining us from and I have the pleasure of introducing you to Jonathan Franco Jonathan is Chief scientist at mosaicamel which recently has been acquired by databricks he leads the company's research team towards the goal of developing more efficient algorithms for training control Networks you might also know him as the author of iclr 2019 best paper lottery ticket hypothesis and in addition to his technical work he's also actively involved in policy making around challenges related to machine learning and today this is the third lesson of our training and fine tuning llms course and Jonathan will teach us about data and data sets for training so this is an exciting topic we're excited to have you Jonathan thanks for being with us thank you so much for having me I'm really excited to be here and share what I know I want to emphasize to everyone before we start there's a lot more I don't know than I do know and a lot more we don't know as a field than we do know so if you came here looking for Clear answers and being told exactly what to do I'm afraid I'm going to disappoint you but my hope is I will share with you the big questions you need to think about the methods for trying to sort through those questions and making the right decisions in in whatever context you're working in awesome so let's go ahead and get started let me get my presentation up and then we can dig in um so I've left three GitHub links here and I'll I'll keep them up I'll bring them back in the future but this is kind of this is my frame of reference these are the code bases I work with every day and all of us at mosaic ml work with every day as we build our models so you know if you're interested in basically doing it the way that we do it and you don't have to there are lots of great libraries out there um this is certainly where I'd recommend starting so to give you a brief overview of the game plan um there are really four things I'm going to cover I'm going to start with a bit of friendly advice um that friendly advice is intended to kind of contextualize how we should think of scientists and then I want to really answer three questions how much data should you use which data should you use and the logistics of getting the data from point A to point B if we cover all of that today then we've been very successful it seems pretty straightforward and easy if only it we're so in practice but you know the goal today is to at least highlight through some of the key questions you should be asking yourself and you know I am going to ask please feel free to you know ask questions I'm going to try to pause for questions after every section here and hopefully we'll get to take a few of them so please feel free to share your thoughts um I'd love to chat you know closer to the content than toward the end so with that I'll start with a bit of friendly advice um this is pretty generic scientific advice it's important though first of all start small and work your way up the thing that scares me most whenever a customer comes to work with us at mosaic Canal is when they say hey I'd like to come work with you and train a 70 billion parameter model that scares me a lot because you don't just go and train a 70 billion parameter model even with all the great tools we have and all the great technology that we have it's never that simple every problem is different and it's very important that you start small and work your way up I want to start with a 125 million parameter model and work my way up step by step and understand the problem better and better spend a dollar before you spend ten dollars before you spend 10 million dollars and it's important to keep that in mind don't ever just shoot for the moon nothing that I'm going to tell you today will tell you exactly what you need to know to just shoot for the moon it's certainly something we never do at mosaic at all friendly advice number two be skeptical of what you read in the literature and I don't say that to mean disregard what you read in the literature but be skeptical think critically the reason is that you know there are really two reasons why I say this um number one is that no paper is perfect especially in an empirical science where knowledge is hard to come by and everything is pretty fuzzy and context dependent it's really convenient when we try to look for simple answers but there are no simple answers and many papers you know we're trained to present simple answers to scientists and kind of Reason two is that our incentives to get a paper published or get a paper to be popular and kind of Advance our careers as researchers are not the same as making sure that we get useful scientific truth they're often to Hype something up they're often to get an idea widely circulated they're often to get through the review process and that means making things simple and clear and making it look like you have a very obvious win when sometimes you don't and so this isn't out of malice on anyone's part but it is important to be skeptical of what you read in the literature that it's true that it's true in the way that it's stated and that it's true in the context of your problem and so the important thing I tell everyone on my team at mosaic and we practice here test everything for yourself don't believe it just because someone else said it and don't believe anything I say today just because I said it test it for yourself Point number three and this is going to sound very repetitive don't trust your intuition test intuition is great for posing hypotheses and coming up with ideas do not try to Intuit your way through it and then make decisions about what to actually do test and measure compare to strong baselines compare to simple ideas compared to things that would be true if your intuition were completely wrong and find out what the data says you may have excellent intuition but if you do you're a far better person than me I have terrible intuition I tend to be wrong a lot more than I'm right and that's fine because I post hypotheses test them realize I'm wrong and improve the last piece here and I think this is kind of you know a little bit even higher level than the other points when people said data science in you know yesteryear five years ago 10 years ago I kind of thought it was a bit of a buzzword the things we're going to discuss today are really literally data science you were working with data and you need to be a scientist you need to post hypotheses test them using experiments run control experiments to make sure that your results are actually meaningful and significant all of that good stuff there are no answers here you have to be a scientist and let the data speak for itself I hope this is helpful framing this is how I think about the world this is how we operate at mosaic ml these are key principles for us and especially in the data realm where we know far less than we wish we knew and you know we don't know much more than we do know this is the only way that you can operate and make good decisions what I like to say about deep learning is that the path to Enlightenment is starting by accepting you know nothing I think the biggest mistake we make as a field is that sometimes we assume we actually know things we don't and much of what we think we know is completely wrong so start from that Zen place of lack of knowledge and build knowledge using science and using data okay that concludes my friendly advice and now it's time to actually talk about the other part of point zero the surprise that isn't actually in in the lecture but you know within the agenda but I do want to cover it evaluation you can't make any decisions until you know what success looks like when someone comes to me and says I want to build a language model what data should I my answer is I have no idea because I don't even know what you're trying to accomplish and even if you wanted to make progress in this problem you would have no idea if you were successful the scariest thing in the world for me is when a customer comes to me pays a lot of money Train's a big model and then turns to me and goes so is it good if that question is getting asked then something went horribly wrong in the process because we didn't set up good evaluation Frameworks for measuring whether our model was good I wish I could tell you which benchmarks to use and I have given you a link here to our Mosaic ml LM evaluation page where we're trying to put together you know where we've shared the evaluation harness and the evaluation benchmarks that we currently use so that you know exactly how we're measuring our models internally and making decisions that evaluation is not a one-size-fits-all process we're evaluating among a bunch of different axes and I can actually show you um you know we're looking at a lot of different kinds of models these radar plots have gotten very popular lately but we're looking at everything from programming ability to Common Sense reasoning to reading comprehension kind of we've taken as many benchmarks as we can find out there that we think are good kind of taken them apart and put them back together into these categories and then looked at our model on these axes this is not the right way to do it but it's a useful way for us and we found it to be productive this will not be the right way for you to build your model because chances are you're building your model with a specific purpose in mind so are we but our purposes are probably different and that means you need to evaluate differently until you have a way to measure whether your model is getting better at the things you care about you shouldn't even begin to try to train a model evaluation is the most important and the hardest part so just as a reminder please you know think about this and I will say you know we have you can access this whole evaluation harness on GitHub this is not meant to be marketing but I do want to put good Tools in people's hands there are a lot of other great evaluation harnesses out there a Luther AI has one that's very useful as well this is the one we use at mosaic you can go to this URL you can go to our llm Foundry repo and go to the evaluation area and you can go and run on any hugging face compatible model um you know you can run this evaluation harness and see for yourself so I think it's it's a great set of tools I hope it's useful to you we it's what we use internally and we wanted to share it with everyone but you know the most important lesson here before you even touch Data before you even begin to think about how you actually build your model you need to know what success looks like and I don't like to work with any customers until they have a way of measuring what success looks like okay that's all for the friendly advice let's talk about data so I'd promised you three questions question number one is how much data how much should you use and there are two steps to any model training process there's the pre-training process and there's the fine tuning process personally I will say I hate the phrase fine-tuning because often fine tuning is not very fine it tends to actually be as intensive or more intensive than pre-training but free training tends to be generic and self-supervised we're just training on next token prediction fine-tuning tends to be much more with a purpose to improve the model on a specific task on a specific domain on something like a chat or instruction following or on a particular language like python or you know a specific an actual human language as well whatever it may be you want your model to learn about a domain learn about a task or both so I'm going to divide how we talk about data into both of these components because they're both very important and I'm going to talk about this in the context of building a model from scratch because honestly I think that's the most fun part of this but you know if even if you're just fine-tuning a model and I say just fine tuning a model there's no such thing as just fine-tuning a model um you're going to still need to use a lot of these lessons and think about how much data and what it will cost you as well I'm really big on trying to measure things in dollars and cents in addition to just pure volume so actually before I show you some chinchilla plots the there are two papers that I've highlighted here because I think they tell two different aspects of a very important story so imagine I said to you I'm giving you a thousand dollars in Mosaic ml a 100 credits GO train the best model you possibly can how are you going to spend that budget and suppose I give you a fixed data set I'm just going to tell you here's the data set you have to use here's the model architecture you have to use but I'm not going to tell you how big of a model to use you could use a very small model or a very big one you could use one billion parameters or 100 billion parameters and I'm not going to tell you how much data to use but I will say I've given you a fixed budget so if you train a bigger model that means it's going to be more expensive for every piece of data you look at so you're going to get through less data if you train a smaller model it's going to be cheaper per piece of data you go through and so you can get through more data so the size of the model you choose determines the amount of data you get to see what is the best trade-off clearly if you have a model with one parameter you can look at huge numbers of tokens but it's going to be pretty bad if you have a model with 10 trillion parameters you might get to look at one token and that's also going to be pretty bad so there's a Sweet Spot somewhere in between here there's a right model size and a right amount of data that will get you the best results that's what that first paper training compute optimal large language model is also known as the chinchilla paper that's where kind of the chinchilla scaling loss comes from that's the question that paper was trying to answer and what previous you know work from openingi was also trying to answer if I give you a fixed budget what is the best way to spend it on model size versus amount of data and this will tell you for any particular budget here's approximately the right size to go with and I'll show you that in a moment now the Llama paper that came out this spring in some sense violated convention because they purposefully chose things that were not optimal they purposefully chose smaller models on more data which meant that for the amount of money they were spending they could have gotten a better model why did they do that like why did they spend their budget less efficiently than they could have because spending your training budget efficiently is not the only constraint smaller models are easier to fine-tune they're easier to do inference on in fact they're even easier to train for a lot of systems related reasons and the Llama authors make the point in the paper that they purposefully chose to train smaller models on more data to do something highly sub-optimal according to that first paper to make it easier to train and easier to work with and easier to do inference on that's a totally reasonable choice and the question just becomes kind of what is the trade-off you're willing to make what is the penalty you're willing to pay from that chinchilla optimal model in order to make sure that other things are easier and I think that's a key question that comes down to choosing the right amount of data so I'm going to show you I believe one and only one graph during today's presentation this is a graph from the chinchilla paper and this is kind of the whole paper in one graph in some sense so let me walk you through this a little bit on the x-axis is the number of parameters in a model so as you get further to the right the models get bigger and each of these points is some model that was trained on the y-axis is the training loss so you know how well did the model do lower is better lower loss means you've got a better model so what I want you to do is take a look for example look at that point that says 3B or 3 billion parameters on the x-axis and kind of trace your way upwards look vertically upwards you'll see how for that same model size you can get a lot of different losses anywhere from a very high loss to a very low loss the difference here is how much budget you spent so each of these different colors each of these different curves is a different budget those numbers in that in that legend of the plot are different budgets in terms of flops anywhere from 6E 18 that's the uppermost curve to 3E 21 which is almost three orders of magnitude more compute and you can see for example again tracing that three billion parameter model looking at that vertical line up and down you can see that as you increase the budget from you know the top curve where it overlaps which is 6e19 um I believe or yeah 1619 all the way down to that bottom curve you're getting better and better loss as you increase your budget that makes sense right you spend more and when I say increase your budget what's happening you've got a fixed sized model so when you increase the budget you're training on more data so one way to think about this is if you look up and down any one of those vertical lines you're looking at that three billion parameter model trained on more and more and more data okay so let's look at any one of these curves and pick a curve you want to look at notice that the curve kind of goes up on the left up on the right and down in the middle what this means is for any given budget for any given amount of money that you have to spend there's an optimal number of parameters that will get you the best loss for example if you look at that second curve from the bottom the three billion parameter model is kind of right around the bottom of that curve that means that if your budget is 1e 21 in terms of flops a three billion parameter model is the optimal model size if you have a bigger model you're not going to do as well for that same budget if you have a smaller model you're also not going to do as well for that same budget and so for the amount of data then you can determine the amount of data by looking at I've got a three billion parameter model this much budget and then you can go back and back calculate the amount of data that you need so this is a really important observation for a given budget there is a right model size and a right amount of data that will get you the best possible results and there's kind of a nice thing that follows onto this and I'm not going to highlight here which is actually there's kind of a fixed relationship between budget and model size approximately speaking what the chinchilla paper tells us is that the right amount of data is about you know for a particular model size is a number of tokens equal to 20 times the model size for three billion parameter model that means 60 billion tokens is kind of the optimal way to train that model now the other important part that you can take away from this plot which I think is really interesting and that's why I've chosen this particular plot from that paper is again let's look at that second line from the bottom with the three billion parameter model is optimal you can also ask yourself how sub-optimal would I be if I were to choose that same budget but have a smaller model let's look at where the 1 billion parameter line comes in you can kind of see that you're going to give up you know about 0.05 in loss which may or may not be significant to you but that's the trade-off the Llama authors were making in some sense they're going to spend their budget some optimally which means the models are going to be worse than they could be for that same budget but that's a worthwhile trade-off for them because they were able to get models that were easier to serve so the Llama authors in many cases were picking a point for any one of these carriers that was off to the left it was not the very bottom point it was somewhere off to the left but to them that made sense and that was a reasonable trade-off and given the popularity of the Llama models that tells me it was also a very popular trade-off that people were happier with if they had done chinchilla optimal models and by chinchilla Optical I mean picking kind of that bottom Point picking data equal to 20x parameter size those models might not have been as popular because they would have been much harder to deploy so this is a really important graph I think in some sense this is the most important graph if you're looking at the question of how much data should you use and how should you choose your data so I want to keep moving a little bit and ask okay so where do you get your data and how much does it cost and the nice thing is for pre-training data there are a bunch of amazing data sets that are out there everything from the pile which was a Visionary data set by the people at a Luther Ai and I say Visionary because they made an open source large scale data set in like 2020 and 2021 back before anybody else was really thinking about this everything up to much more recent data sets like the red pajama data set from together AI or the Falcon refined web data set from the folks who made the Falcon models the star coder data set which is a data set of prop of permissively licensed code which we found really useful at mosaic and the dolma data set that just came out from our friends at the Allen Institute for AI which looks really promising these data sets are big they're free you can download them and you can use them to build models these data sets mostly contain web scrapes and web scrapes tend to be useful there are a lot of tokens and I just mentioned to you you know we we talked about you know if we go back to that previous slide um three billion parameter model we said 20x so let's take the 70 billion parameter llama 2 model for example the you know the optimal amount of data there is a 1.4 trillion tokens that is a huge amount of data I'll go through in a moment how big some of these data sets are so you can get a sense for kind of what the trade-off is but that is a massive amount of data you need probably a solid 1 trillion to 3 trillion tokens to train a modern large language model and the only place you're going to find that as of right now is scraping the web that's why we rely so heavily on web scripts right now we supplement with all sorts of other great data sets out there Wikipedia GitHub what have you but at the end of the day you kind of need the whole web and the web is massive compared to some of these other data sets I'll show you that in a moment now I also want to talk briefly about after training data and I'm going to highlight a few things here um first of all I'm going to highlight how much it costs now after training data I'll give you you know just an example so you kind of know what I'm talking about here once you've trained a model that's not enough you can think of like the original gbt3 where you had to really prompt it carefully versus say chat gbt these days where it just follows instructions and the way that you get there is by fine-tuning the model afterward on instruction following tasks you literally give it instructions and responses you know write me a poem about deep learning in the style of Shakespeare and then an example of a poem or write me a program that sorts a list and then a Python program that sorts a list you give it a lot of examples of that sort of stuff and I'm highlighting a few things from the Llama 2 paper because they did a fabulous job the authors of that paper in terms of highlighting what you actually need to do they said that we found that sfts which is supervised fine-tuning and in this case instructions and responses annotations in the order of tens of thousands was enough to achieve a high quality result we stopped annotating sft after collecting a total of 27 540 annotations so those are you know I've kind of put some rough costs here these are costs I've gotten from you know I've heard from friends who are in the field who are collecting this data this is if you work with data labeling firms that do a very professional excellent job on this it's about 30 to 50 per example there are some open source data sets but they're a little bit spottier and the licensing on those data sets is a little bit more questionable because a lot of them were created by scraping from gpt4 and you know there are terms of service that may or may not allow that for gpt4 so the first part here is that you generally do kind of the supervised fine-tuning or instruction following I also mentioned at the bottom a multi-turn chat conversation you can imagine instead of one back and forth you have let's say seven total you know one person says something you know you and the and the model going back and forth that tends to cost on the order of you know from what I've heard from friends about 100 to 200 per conversation so again this is quite expensive you know about 30 000 annotations times about let's call it fifty dollars and you know you can do the math that's 1.5 million dollars right there that meta might have spent here now there's something interesting that meta also said um in this paper that I think is worth highlighting surprisingly we found that outputs sampled from the resulting sft model the model that was fine-tuned on this 27 000 example data set we're often competitive with sft data handwritten by human annotators suggesting that we could re-prioritize and devote more annotation effort to preference-based annotation for rlhf so I did not work on this paper I have no inside information but what this indicates to me is probably they used a lot more than 27 000 examples but they had llama 70b generate more examples because they found that that worked pretty well and they could kind of have humans curate those examples get a little human input and it was much more cost effective from there so it's something to think about and a lot of these models like llama2 have licenses that restrict you from using the model to generate data to train other models and I think it specifically because you can now do this kind of thing now there's one other piece to doing after training data and that's rohf or reinforcement learning with human feedback the idea here is that you know there's having instruction following but there's also just trying to make sure that the model produces outputs that humans find helpful find safe that humans prefer and the way that you typically do this is you provide two different outputs of the model and you have a human say I like this one better than that one or I like that one better than this one there are much fancier ways of doing it but that's kind of the high level idea and you know meta did this iteratively they took their model they fine-tuned it then they did some rlhf they did this pairwise comparison updated the model based on that took that updated model did another set of human feedback did another round of this and did this a bunch of times and in total they collected 14 batches of human preference data on a weekly basis consisting of over 1 million binary model generation comparisons at five to ten dollars per comparison you can do the math that's pretty expensive and so when I think about cost of data not just kind of right amounts but cost this is kind of what it takes to build a large-scale general purpose model now it's important to note not everyone and hopefully you know almost nobody who's listening to the me talk right now needs to build a general purpose model so you know a lot of people are trying to build very specific special purpose models and with that in mind you need a lot less data on both fronts if you're fine-tuning an existing model you need a lot less data than this but it is worth making sure you keep in mind your data budget or where you can get data that looks like the actual interactions that people are going to have with the model that could be existing data that you have from a previous model or previous setup that you have it could be that you go and actually procure that data that you have some other data that can kind of fill that need but it can get very expensive yeah we have a question from YouTube how useful can data augmentation be in expanding out these data sets can you make a one billion token data set reliably into a three billion token data set with augmenting or not that much um so data augmentation is a tricky topic because what I just discussed in some sense was using a model to generate more data for kind of instruction fine-tuning or sft supervised fine tuning now data augmentation for pre-training data is something that hasn't worked very well as far as I've seen I haven't I've seen some papers that discuss it and proposed ideas but I don't know of anyone who's using it in practice right now um so no I wouldn't rely on data augmentation at the moment in computer vision and I come from the computer vision world originally data augmentation is very very useful but I think the answer in computer vision is that when we have an image we have a lot more information intrinsically about that image in most contexts in the real world we can left right flip things and it doesn't matter we can crop things differently and it doesn't matter it's much trickier to think of something equivalent to that for text you can't kind of Left Right crop a sentence or left right flip a sentence or crop a sentence or change the aspect ratio of a sentence so text is just a much harder to work with in that respect than images images we just kind of know a lot more about as humans and have a lot more priors we can impose to kind of take a small amount of data and turn it into more data it's a great question but as of right now we don't have a good way to do that so I want to pause here for any other questions on this first section before we move on to the second section of which data you should use so um you know darker Taylor um if there are any other questions that folks want to ask Now's the Time to get them in we have a couple of questions also from visible so the first question is on chinchillas getting close and it is the question is is it for a fixed architecture what about same number of parameters but different model types so it is mostly for a fixed architecture they're kind of scaling rules that we generally use to to create standard style one billion or three billion or 7 billion or what have you parameter models as you play with the architecture though the scaling laws probably change chinchilla is still a good rule of thumb as a starting point and it's very expensive to go and compute your own scaling off we kind of go back to that graph look at the number of points that they have and imagine going and doing that before you train a model you'll spend orders of magnitude more compute creating these scaling laws and you will actually training the final model so it's not affordable for all of us um you know even at mosaic certainly not you know and we have a lot of compute available the Deep line folks put in a lot of time and money into this so the answer is as you change the architecture the scaling laws almost certainly change a little bit at least but chinchilla is still a really good rule of thumb as a starting point and find a good starting point is the whole name of the game here beyond that you do have to do the science to figure out what makes sense for you a great question thank you that is also a question about synthetic data do you see that useful for pre-training at all it's a tricky topic it's not something that I've used personally but I do see a lot of people talking about it more and more um but I'm still pretty skeptical because at the end of the day where does the synthetic data come from it probably comes from a model and I you know this isn't scientifically back this is intuition so take it with a complete grain of salt and disagree with it but I'm not a big believer that you can just create something from nothing that many of the strategies that seem to work best for using a model to create more data involves some human curation or human involved everything from having the model generate a couple of candidate examples and then having a human choose the best one or edit you know one example that they prefer even things like constitutional AI with which anthropic advocates for where you have the model producing example then you give it a principle from a constitution that a human is written and ask the model kind of update this output to make sure it's in line with this principle that's still kind of using human input and so I think the art is in reducing the amount of work or kind of using each human hour as effectively as possible but I'm kind of skeptical of the idea that we can simply create something from nothing and synthetic data is more along those lines without some kind of human input or human curation but again that is just intuition so take that with a huge grain of salt thank you I wanted to go back to your comment about evaluation and the fact that you should start with figuring out like what you expect this model to do and to mentioned some of the benchmarks that are typically used to evaluate llms but I wonder for companies that want to train their own llm and by the way there is a question like when you should think about training your own llms such as using one that is already available but if you if you want to train one for your use case like do you have any advice how to think about evaluation yeah so evaluation is the hardest part of the whole thing the gold standard of evaluation is using humans or putting it in a real world setting and seeing how people interact with it I think code completion things like GitHub pilot are a great example of this you can put out a model into a real world setting and see how many people press Tab and actually accept the result of that that tells you a lot about how well your model is doing and whether you need to do better so the nice thing for a lot of companies especially is you're typically not putting this model in a brand new use case you're typically integrating this model into a flow that already exists and this is augmenting humans or this is automating something and you I hope you already have a way to measure success of that use case in general in which case you can look at how the model impacts that that result or how you're measuring humans today or how you're measuring the existing process of the existing model but when it comes to evaluation it is incredibly bespoke and Incredibly tricky and it is much more art than science right now I would say there's almost no science I will also caution people there are a lot of publicly available benchmarks there are things like Helm and mmlu I think they're all terrible I think every single public evaluation that we have right now is terrible um I don't trust them as individuals I don't trust them as Composites I don't even trust their own Mosaic evaluations to a certain extent I think we're doing the best we can and it's okay to do the best you can but that doesn't mean you should take any of this is the gold standard or you should look at a leaderboard and say ah this model is better at home than that model therefore it's a better model that's just not true these are pretty bad benchmarks you should go before you ever rely on one of these benchmarks go actually read The Benchmark they're generally not very long you can look at a few questions and ask yourself is this a good test of what I want the model to do the answer is almost certainly going to be no unless your model is answering multiple choice questions or answering like questions about physics or what have you um it's almost certainly a pretty bad test of what your model can do relative to what you want it to do it's the best we can do but it's the reason why you should be building domain-specific evaluation and you had mentioned you know that one other question of when should you build your own model um and the answer is really that first principle I gave you all the way back to the beginning I'm even going to scroll back to it just you know to put it out there start small and work your way up by which I mean if an existing open source model solves your problem effectively you're done go home take the win be happy if gpt4 solves your problem and you can use ppt4 and the cost is okay take the win go home worry about other things work on harder use cases the folks who come to me either can't use gpt4 or gpt4 doesn't address their problem or an open source model doesn't address the problem and when they come to me the first thing I tell them is go fine tune an open source model and see if that works fine tune one of ours fine tune llama fine-tune Falcon what have you if that solves your problem good take the win go home you're done we get to pre-training when none of those things work and there are a lot of cases where none of that stuff works either because you've got a very specific domain you've got a lot of data what have you but work your way up climb the ladder start small don't go straight for the big stuff right away you'll know if you need this any other questions at this stage should we move on to the next section I think let's let's move on and we'll we'll still have some questions we can keep them to the end if we if we still have time okay so the next question is which data and I wish I had a good answer for you on which data um they're going there are more questions and answers here but I am going to tell you kind of what exists and how we think about it and I'm going to start by giving you a bit of an exercise I'm actually going to be quiet for one minute and for those who are watching get out of sheet of paper or think to yourself what I've done here is I've taken a table that we put in the MPT blog post here are all the data sources that we had and I'm going to go through them with you one by one just so you become familiar with them what I've hidden here with these question marks is What proportion of the data set to you or What proportion of our overall data mix should come from each of these data sets there's no right answer here but I think it's a fun exercise to have to think this through on the left you'll see that there it starts with three different common crawl data sets three different web scrapes mc4 which is a big multilingual although we've just worked we've just picked out the English portion here but it's a web scrape and it's actually web scrapes going back over several years and it's 2.4 trillion tokens that's a lot of tokens we have C4 which is one web script for 2019 about 100 billion red pajama common crawl a more recent one about 900 billion tokens then we have a few other things we have the stack selected languages this is from the stack data set from Star coder it's a bunch of languages from GitHub we focused on the common ones like Python and Java and C plus um Wikipedia from the red pajama data set this is just a scrape of Wikipedia about 5 billion tokens um the markdown component of the stack this is like readme's and things like that so it's more text than code about 100 billion tokens semantic scholar is now a data set known as pesto it's about 50 billion tokens of Open Access scientific papers the books data set as red pajama prepared it a scrape of the latex from archive is red pajama prepared it and stack exchange is red patronum this is these are all the inputs for MPT models and you have some tough choices to make here I want you to go and figure out which how would you get to one trillion tokens you could get there really easily by just using mc4 and using a web scrape but that's you know the web scrape is okay it probably has a bunch of HTML that hasn't been stripped out of it it probably has a lot of advertisements and low quality Pages you could also get there by going through Wikipedia 200 times that's a lot of repetition but Wikipedia is a pretty high quality data set you could go through the stack selected languages you know twice that's a lot of code though do you really want a model that just does code so I'm going to actually pause for 30 seconds I want you to think about this this is an exercise where if we were sitting together in a room I would ask you to write it down and share with me um but I'm curious for you to do this and you know I'm not sure how interactive we can be here dark but I'm kind of curious if anybody wants to you know share what they thought I'd be really curious to hear responses if anyone wants to post it um so take 30 seconds and kind of think about it for a minute I love this exercise let's see if there's any any response on our stream at no pressure but I'm always curious if we're in a room together I would definitely be calling on people and asking them you know what do you think maybe I can I can try myself like yeah yeah I'm curious what would you do here I I feel like a lot of deep learning is is being like a little bit of a magician and ingredients and trying to see like what comes out I think the challenge with like this llms is like it's very hard to do a lot of experiments because they are super expensive so um I would um I would say um I would probably like do more of the high quality data sets um I heard a lot about code being important so I'd probably like use a lot of GitHub or um all the stock I don't you're not being specific enough I want percentages here give me some percentages on some of these data sets talk to me okay yeah I don't have the numbers worked out but um I would um probably use like 60 percent of like the different uh web data and I don't know how to divide between them I need to to to learn a bit more maybe 20 codes and um 10 percent um Wikipedia semantics scholar books and um yeah yeah ordered like 15 and 15 like 15 like the all of the let's call it like the high quality data sources uh okay so I mean I made up my mind so 70 would go for web uh in general um like a mix of of of of what's that uh 15 would go into like the different code data sets and 15 would go into like books Wikipedia archive like all of this higher quality sources why that balance like why only 15 code and not 50 code why only 15 high quality and not more like talk to me a little bit walk me through your your decision here um I yeah it's it's very much intuition um now that you're asking like I well actually I think like that that isn't maybe that much code like it would like there would be a lot of repetitions like I need to look in detail of the numbers but I assume like if we if we do more percentage of code like we would need to do like many epochs over the same code you actually have plenty of code here so you've got if I wanted to come up with a trillion tokens and you've got 463 billion tokens of code so you could just go through all the code twice and be fine um there's it turns out there's a lot of code so this is you know I'm being unfair to you right now I'm being unfair for a specific reason um I didn't even tell you what kind of model we were building or why um if I told you we were building a code model you'd probably answer very differently than if I told you I was building a model to be good at trivia so you know again this is a great moment for me to repeat to everyone and kind of you know make fun of dark a little bit on this evaluation evaluation evaluation you need to know what you're doing before you can even start to ask this question um but with that said you know I'll show you what I came up with um these are the proportions that I chose um so I went with about you know honestly pretty similar to what you chose I went with about 70 of the web calls went with about 10 code and then the remaining 20 or so was spread over these kind of higher quality data sets if you look in that rightmost column in the table this is kind of a cool part you can look at the Epic so how many times would you go through each of these data sets you can see I only I only went through 15 of mc4 but I went through Wikipedia eight times and we do tend to see that going through data sets too many times you do start to you know get diminishing returns we saw Beyond four to eight times of going through the same data set you see some diminishing returns here um this is not to say what I want to emphasize to you do not look at this and say Jonathan knows the answer I chose this late at night after trying a bunch of different things and kind of giving up and saying ah I don't have the quality of evaluation to tell the difference we've gotten a lot more sophisticated since then at mosaic and you know I would do things very differently going forward we're trying to make some choices for some new models that we've been cooking up lately and you know the same applies like it's it's a tricky thing and the real answer is know what you're evaluating on and then try to explore which data sources seem to work best for what you're evaluating on I wish I had a great answer and I could tell you what to do this is also this is what the Llama folks did in the Llama one paper um and I assume they didn't do anything too differently in the Llama 2 paper you can see kind of similar about 82 percent common call in C4 that's web scrapes only 4.5 GitHub and we did find anecdotally that our MPT models were better at coding than the original llama models and you know so this does matter a little bit and then you know about the remaining um you know 15 or so from these other higher quality sources like Wikipedia books stack Exchange um so notice that I've used intuition a lot here there are a lot of key questions here that really matter um first of all should you even mix data at all or should you just say fresh tokens are better just if you have a new token use it and whatever the proportions are they are if Wikipedia is only you know 0.5 of your data sets it's only 5 billion tokens that's fine you only see Wikipedia once maybe that's the right answer now obviously this doesn't make sense in the limit um you know if you have a lot of very low quality tokens that probably seems bad but again don't trust your intuition trust the data and so kind of this next question quantity or quality I think dark you and I both aired on the side of qual of quality we wanted to make sure there was some quantity but we did way over emphasize some high quality data sets because intuitively we must thank high quality data like Wikipedia that's more important maybe or maybe not test it find out find out what matters in your application there's no reason to trust our intuition here so test it and test it on a small model you don't have to test it on the full thing and train 10 models try it small the beautiful thing about chinchilla is that as you increase the size of your model you should also increase the amount of data so the cost of training a bigger model basically increases quadratically which means it doesn't cost you that much to train a lot of small models before you train one big one and so you can actually do a decent amount of experimentation and upload your budget um there are questions about whether you should de-duplicate so I'm going to dwell on deduplication for a moment I'm not going to tell you what to do but I do want to highlight the questions that are out there and the answer is again try it for yourself there's this really seminal paper by Catherine Lee and colleagues at Google brain at the time um on deduplication but I'm actually going to pull some lines directly from the abstract here because I think it's so well written and it says everything I wanted to say but better we find that existing language modeling data sets contain many near-duplicate examples and long repetitive substrings and this actually has an effect as a result over one percent of the unprompted output of language models training these data sets is copied verbatim from the training data so the the duplication of these sequences leads to memorization now I think it's important to say um you know specifically what they were looking for we developed two tools that allow us to deduplicate training data sets a lot of these tools are now their many versions publicly available to do this for example removing from a C4 a single 61 word English sentence that is repeated over 60 000 times so I think the key takeaway from this paper I think a lot of people read this and say oh we need to de-duplicate everything we can't have any duplicates that's not what this paper is saying this paper was finding things like marketing copy and you can look at some of the examples here um or other kind of form text that was repeated tens of thousands of times in the data set and removing those tended to lead to better results for the model and less memorization and also saved you some time because why should you see that example sixty thousand times but it's important you know don't overread into this this was about highly repeated examples this is not about something that's repeated twice and it's an important distinction there now there are much fancier did a due duplication techniques that have come along in the time sense um these include looking at semantically related text not just identically or near identical text so things that where you you embed your data and then you look for embedding similarity and remove things that are too similar because they're probably near duplicates this is you know this is another fancier method that can push it even further now what I will say is that for all this due duplication I would say the jury is still out and there'