PyTorch Expert Exchange: Adapting open source models with Open-Instruct and Tulu

Key Takeaways

live all right awesome uh Hey folks uh Welcome to our second uh talk for the py expert exchange series uh today we have Hamish from the uh from AI University Washington lab who will be talking about uh adapting open source models with open instruct and Tulu uh so if you're tuning in live on the YouTube channel please feel free to ask questions in using the chat feature and we'll uh take those questions after the talk um with that I want to hand it off to Hamish um feel free to go from here cool um thank you for the intro um yeah so hi I'm HH uh I'm a PhD student at udub I also work parttime at ai2 um and yeah I'm going to talk about sort of open instruct Tulu which is sort of two uh efforts going on interl efforts going on at uh sort of a ai2 mainly but also with a lot of udub folks involved um specifically I'll sort of talk about open Touch on open instruct and touch on um Tulu 1 through three uh which are sort of a series of models that we've been building I'll explain more about what those are bit in a bit um and hopefully sort of cover what like building a modern posttraining pipeline or adaptation pipeline for language models looks like what are the sort of components of such a pipeline and touch a little bit on sort of how those needs differ to needs in pre-training when you sort of want to do pre-training of language models um some like quick disclaimers um there's sort sorts of like some ai2 specific Integrations that influence some design decisions and stuff for this so don't worry uh like that meant that we made some of the choices that we did um there's also like I'm going to be talking about open instruct which is one particular sort of repository for fine tuning but there's lots of other great ones torch tune uh TRL Axel are all sort of like well supported and support I think prettyy similar features to us although in like slightly different ways um and I'm probably going to remain a little bit higher level but I'm happy to like answer questions getting into like much finer details especially towards the end um cool but first before I like get into like open instruct and Tulu like why do we care about language model adaptation in the first place um and you know what is it typically uh in the past language model adaptation has looked mostly like instruction tuning so we take uh a bunch of inputs that have uh sort of instructions of like what we want the model to do and then some like query uh train and train the model to Output the answer and if we do this over a whole bunch of tasks turns out the model can sort of generalize to new tasks that it hasn't seen before um as well as sort of like adapt it to sort of like be a bit more like a chat interface or something that looks more like a chat GT chat GPT or clae that a lot of people use nowadays um more recently in the sort of past like one or two-ish years this adaptation pipeline has gone a lot more complicated with things like RF there's lots of like people doing stuff like synthetic data um whether you do some rejection sampling as well often people sort of do multiple rounds of like data annotation and iterations um and so I think like nowadays there's sort sorts of a lot of moving parts and things that you want to be able to do when you addap models and I'll sort of get to that um and what adapting models lets us do is basically lets us use them in a way that uh we prefer so if I go and ask uh llama 3.1 you know what is 2 plus two the output I get from the base model is uh some talk about roofing contractors and it doesn't actually answer the question but if I go and use the instruct version which is the sort of adapted variant then it just sort of plainly gives me the answer which is why I'm looking for um similarly adapting models partly because of this e of use and also for other reasons sort of lets us get more performance out of these models so if I take llama 27b um and I do some instruction tuning on it then its performance can reach almost the level sort of llama 213b out of the box and likewise if I then instruction tune llama 23b it also improves significantly um and this is sort of like the the generally the case across like all sorts of models model sizes and different model families um at ai2 the sort of main repository that we use for doing all this sort of adaptation like instruction tuning RF training and so on is called open instruct um it's a public repo we work on it in public it's permissively licensed um and so you know you can go and look at it now if you want to um and the primary output of this model is a sort of model series that we call Tulu uh the aim of Tulu is to sort of combine existing open resources to build a like open reproducible general purpose model so we take good base models that exist out there um and try to work out strong recipes for building sort of general purpose uh models that behave similarly to what you might see uh in more in Industry coming out of Industry Labs or industry products um but we share all the details we share exactly what data we use we share what hyper parameters we use we share exactly what code we use um the reason the model is called Tulu is because it's sort of a child of many camelids uh we started this work originally when sort of like the first llama models and sort of like the first sort of wave of instruction tune llama models came came out when we were trying to work out uh you know from how we could combine all these different data sets that were coming out into like a sort of strong uh Baseline for then doing further research on top of uh if you're particularly curious the Tulu series is sort of captured in these series of papers I'll be touching on them a bit as I go uh but I also recommend you go and read them if you want uh sort of more details and sort of really to get in in depth of of what we did for these models so to get to the sort of core of this talk I want to talk about like how the Tulu series of models has improved over time and how we've sort of uh changed open instruct and sort of how open instructors supported that and the sort of first big thing um perhaps not too surprising is basically just like doing lots of data mixing and improving our data mixtures so when we started Tulu uh the first sort of Tulu project was basically we just took the Llama models llama one at the time that were available and then caded a set of like instruction data sets that we thought were interesting uh and cover covered sort of a varied sort sorts of types of data so like synthetic data human annotated data sort of more traditional NLP data set data sets Etc um and wanted to sort of you know see how well they did and to sort of like take a chart from our original paper here uh you know maybe this is soat obvious but different data sets were particularly good at uh particular things so like coding data were you know better for coding and uh your more sort of synthetic chat oriented data sets did better for chat and so on um however if you know you were slightly careful about the mixture the mixing the data together uh and balancing your data your sort of data sources reasonably well you could then uh make a data mixture that sort of captured most of the benefits of training on any par one particular data set uh without getting the downside so this sort of bottom row down here uh is sort of doing the best on average and generally speaking is always improving over the base model uh in terms of how this looked like behind the scenes uh originally it was uh we had basically some fine tune scripts that let us like submit sweeps across uh different data set files uh on some ai2 internal compute once that was done we'd then uh run some eval script that automatically kicked off a set of evals uh that we would then sit down and sort of manually annotate into a spreadsheet which I've captured on the right here uh this spreadsheet goes for like thousands of rows and there's a lot more results in here um I think we had to retire at some point because we're just kind of too slow to load uh and we were pushing Google Sheets a bit too much uh nowadays we've sort of improved this experience a bit in open instruct um one particular big thing that we've done is allowed sort of data mixing directly in config uh so may relying a little bit on Hing face data sets as well as some ai2 stuff you can basically Define exactly uh how you know what data sets you want to load how many samples you want to load them in or alternatively like what percentage of data uh you want to load in uh submit a training script that automatically runs that then automatically kicks off a set of evals that then gets uploaded into some into an internal leaderboard that we have uh this means that basically all uh a researcher has to do is sort of make some data that they think is interesting um and then they can and then kick off a job and then the training the evaluation and the comparisons are all sort of like automated to some degree they can you know once within sort of usually for smaller models these jobs take uh training takes like within a day so sort of within 24 hours you can go from an idea with a data set to sort of a final eval result um and comparisons in Fair settings with other models that you've tested uh this is also nice because it allows us to track our sort of data mixtures uh in GE which is a little bit more principled than using a really big Google spreadsheet um and so that's sort of like the first thing it's maybe I I think like having e easy data mixing and U making it easier and easier to sort of just randomly try out in incorporating new data and updating our data mix over time is really important and so to sort of like highlight this uh here's uh a sort of I'm going to sort of like track the performance of this model over this talk so if we take llama 38b uh base and we train it on the Tulu two mixture which is a mixture that we came up with in November of last year uh and evaluated it across a sort of set of evals uh we get roughly 50% performance if we just took some time to update that mixture with some recent data sets that have come out we can sort of bump that up to 53% um without sort of doing you know without sort of doing too much extra work right no new fine-tuning techniques or anything just sort of trying out some new data sets that have come out and are publicly available um the next sort of of thing that I also sort of already touched on is sort of supporting varied evaluations so if I go back to this sort of table that I showed before um like an important thing that something that was really important to us is that we were evaluating these models on uh a pretty reasonable Suite of evals that covered uh capabilities that we cared about so we cared about how well they were doing it like math and reasoning we cared about multilingual uh you know multilingual performance we cared about coding and we cared about sort of like chattiness type uh abilities with our packet eval um since then we've sort of expanded the amount of evals that are available in open instruct and we've tried to keep uh the ability to add eval sort of as light as possible uh evaluations basically are like a a python script that sit in a folder maybe with there's some light shared utilities to make things like you know uh calling a generation Loop much easier for example um and this allows us to sort of have lots of uh different types of evals in there there evals that are using llm as judge setups there evals that are sort of measuring pass at K across multiple Generations there's evaluations that are using classifier uh models trained to be classifiers as judges um that's really example for useful for example in doing toxicity uh evaluation there's also more standard multiple choice evaluations um and this has been really important and also like uh traditionally kind of a sticky issue with some of the pre-training folks where often the type of evaluations that we care about in the adaptation side a lot more free form and a lot more about uh getting models to sort of generate long bits of text and then uh evaluating them either with other models uh or in some other structured way rather than these sorts of the more sort of multiple choice or rank Choice type tasks that you see in pre-training um we've also as I said got this sort of like leaderboard that we continuously add uh and update the evaluations and nowadays we also sort of like track model made metadata such that it's easy to if we want to add an eval rerun all of our existing models on that new eval to get comparisons or if we find a bug in or a new prompting setup for an eval that uh we think is better we can then swap that in and update and update the leaderboard um there's also lots of fun filtering options and stuff and I know uh recently A lot of people tell ai2 have probably spent a lot of time sort of just like sitting with this leaderboard open uh and and poking at it to get ideas of what data sets have worked particularly well for what evales or what sets of EV vals um and I think like it's really important to cover a a broad spectrum of evals because a lot of these evals are either like particularly uh narrow like obviously GSM is only covering math but even the sort of more broader evals can be biased in in weird ways so when we were uh working on the Tulu One models for example we found that our packet eval one uh had a very striking sort of length bias if you plot the sort of win rate of alpaca 1 and the average output length of a given model it forms almost a a disturbingly straight line This is being somewhat fixed with our Paka Val 2 to some degree but it it actually kind of still exists a bit I found um however even if we did like you know get humans to sit down and uh judge our model outputs uh humans themselves can be biased in all sorts of interesting ways uh I like this sort of example uh where this paper sort of suggests that uh humans find more assertive model outputs to be higher quality um and you know maybe you care about this like this is this is this might be good if you just want your model to sort of sound good to a person but if you care about the actual sort of reasoning or uh uh reasoning capabilities of your model or the sort of like factual knowledge capabilities of your model then I think it's really important to uh include and design evaluations that are specifically targeting those um likewise if you only look at one particular evaluation it can uh paint a misleading picture of how well your model is going um so for some recent work we looked at uh training the Tulu series of models with po um and we found that as we train the models for longer with po so that's the sort of x-axis here is the number of poo steps uh they continue to just improve on alpaca eval to alpaca eval 2 just kept going up um until we stopped training eventually uh however the model could just work worse uh at most of the other evaluations present so it's uh instruction following score if EV Val score dropped and it's a GSM uh grade school math score also continued to drop uh and so this is like a sort of another reason why we sort of care about like continuously expanding the scope of our evals and swapping in new evals where we can uh to order in order to get sort of like a good idea of how well our model is doing as a generalist model as opposed to just as sort of like a chat assistant or just as sort of a math model or and so on so considering wider varieties of evals and making it easy to sort of just add add more or update existing ones where possible um and there are some evals that we that we're tracking particularly a lot right now if I go back to sort of the table I had before um two evals that I I kind of have been very interesting for me to track a sort of math which is harder math problems and Al packet eval to which I've also talked about a bit um and here you can sort of see that you know again we balanced our data mixtures and we got some improvements uh but uh and we're sort of like in when we're designing our data mixtures sometimes we decide to Target uh one or two of these evals in particular like trying to gather math data to see if it improves our math scores and so on um but I've been talking a lot about just like data and eval right now uh I haven't been talking about the sort of sub stuff that I mentioned earlier which is sort of preference learning and RF the sort of entire Center of this big chart uh that I showed earlier um and adding sort of preference learning has been uh and RF in general has been we have been working on this adding these to open instruct in a variety of ways um we started by just looking mainly at DPO and poo if you're not aware of what these methods are PO is basically uh an online sort of RF in general works by training a reward model on human uh preference data and then training your language model to try and produce uh outputs that get high reward in some sort of RL Loop DPO and a sort of range of Al uh direct alignment algorithms instead just sort of train your model directly on that data um using a variety of different methods uh adding sort of DPO or poo into your pipeline usually results in improved performance especially if you care about things like our packet EV Val or toxicity um so when we added it to the we uh added DPO For the First Time In the tul2 Mix uh to the tul2 models sorry and we saw really big gains in our packet eval we on average like the models did improve a bit apart from the largest although this is kind of due to uh multilingual performance dropping uh we didn't train on multilingual preference data so this is a likely reason why um and so of since then there's been like a veritable explosion of dpos esque uh algorithms out there uh that all claim to be uh really good in one way or another uh here's sort of like a nonexhaustive list of these from the recent syo paper um and I think there's sort of more coming out all the time and in open instruct we've added a bunch of these we have DPO as well as a length normalized version we've added uh simpo and wpo we've added some code for rejection sampling um and we have uh I don't know if the pr is merged yet it should be merged this week sort of online training as well with Po and online DPO so there's sorts of if you start with an instruction Trent model there's all these sorts of different uh preference learning methods that you can try out um for your post training uh and this has been uh really important for improving certain evaluations is sort of like the third thing so if I I sort of go back um to uh oh actually and one other thing that I forgot to note is that often we also uh currently I don't think we have a script committed but we've also found merging these models to be particularly effective as well so sort of taking say a DPO train model and a model trained on some particular data set and merging the two to get a sort of combination of the abilities um so if I go back to this sort of chart here um we improved our data mixture a bit and this improved performance a bit we wanted to get better math performance so we actually trained and model on a math specific mix and then just merged it with the original model and this boosted our math scores by quite a bit and overall improved performance a little and then we did some additional DPO training and this really boosted our packet eval 2 uh while uh not dropping like performance elsewhere too much and so resulting in a sort of overall and improved model um so to give you like an idea to sort of graphically lay this out we sort of started with llama 3 we trained it on this Tulu preview mix we then trained a math specific model merged the two got a overall better performing model and then did DPO to uh further improve it and this is honestly a kind of like more simple model lineage of some of the models we've been training internally recently I've seen cases where we've trained some model on some data then DPed it then merged it uh then trained on some more data and then DPed it again um and I think sort of embracing this chaos has been uh quite important to uh work out what methods work particularly well and what and sort of hill climbing on the evaluations that we have uh so if I go back to this we've sort of now internally reached a point where we're sort of roughly at the level of llama 3.0 instruct with a good recipe and we have a little a bit more to go with evals in evals like math and so on um which we're working quite hard on cool uh I'm just going to quick skip over this um but I also note that often times with these sort of especially with these online training methods we often see that generation times are the real bottleneck um and so making smart use of uh libraries like VM has been really important in keeping our sort of Po training times down as well as generating synthetic data from larger models as well um and finally I'm just going to quickly uh talk a little bit about going Beyond Tulu so the open instruct code base is used at ai2 not just for Tulu it's also used for fine-tuning Theo model uh most recently it was used for Theo model which is a very cool Moe that Nicholas just put out uh there's a few fine tuning results in there that were uh trained with open instruct and you can go try out that model um we also find it useful to be able to take base models that other teams have put out integrate it into our pipeline um in order to sort of compare how well other teams instruction and adaptation pipelines are to the recipes that we've been developing uh and so this is a final thing that's really important for AI 2 uh is that we want to be a is that we really want open instructor also be flexible Cross or varied model families I would probably say overall we sort of lean a lot more towards flexibility than speed optimization our data sets are much much smaller than pre-training data sets we have a much bigger sort of bag of methods that we like to use uh with all these preference learning me methods and and rejection sampling and so on um and so we always sort of lean towards flexibility over like optimizing stuff particularly hard um so like we're trying of course like this isn't the end for overd truck we're trying a lot of new things uh a lot more things we're trying we're sort of trying to get more and more into online training so further developing the sort of Po and uh uh online DPO toolkits with things like process reward models and so on uh We've also been looking at like trying to make sure that we uh we're not hill climbing by just including our test data accidentally which could happen when you do a lot of data mixing there are some popular data sets that have crazy high packet eval contamination out there check your data um as well as thinking about like data filtering techniques in general you know um how can we isolate cases where there might actually be good data but it's wrapped in a data set that has a lot of other uh useless or noisy data um as well as a ton of other stuff and yeah uh thanks for listening um I'd like to especially shout out Yong Wang and Hannah uh who Yong worked uh as sort of my was my primary first author on most of this work um and Hannah is my PhD adviser who's also advised all of this stuff uh uh and has been really helpful uh and if you're interested in the code uh there's a sort of QR code up here and a sort of Link uh and feel free to reach out uh on X or email or if you're interested in some of these works you can probably find them on my website yeah and that's about it thanks awesome thanks a lot heish that was a really interesting talk and and great to see how uh open instruct has progressed over time uh for like post training optimizations of the llms um I guess like I had like one very high level question since we have a lot of members here from the pych team I'm curious about like how you use like what features of py you really find valuable and what if anything you think you know pych could do better to support um your research yeah I think um so we lean a lot on like a lot of like various open source libraries and codebases out there like Huggy face pytorch uh we lean on quite a lot I think like as I was saying especially in sort of like with these like online trainers um and like VAR flexible options I think the ability to sort of like have something that looks like a self-contained little script that you can very easily jump into and debug and hack on is like super useful uh we did do some like Po experiments in like other sort of more jit Frameworks like Jack and stuff and debugging these was always like way way way more painful than say like in a simple uh pie torch Loop or something like that which is nice sort of dynamic uh computation graph stuff um the other thing I feel like um probably we actually might like look more closely at torch tune and stuff in the future uh we've been one one thing is that like we've been relying a lot on deep speed mainly for like getting up to larger scale like training larger models at really large scales um but I think there's been a lot of like very nice features that have been added into pytorch recently um for sort of improving scalability here that uh hopefully we can kind of integrate one thing is that like as I sort of mentioned we' like to not tie ourselves too tightly to any particular um model architecture choice because it's nice to be able to take you know a model that sort of like someone has just put out and like quickly try it out put it through our Pipeline and stuff like this um but at the same time there are also models that we lean on quite heavily like the Llama series for example um where it does sort of make sense to try and like potentially to optimize that a bit more yeah thanks so much Hamish for for a really interesting talk so I I maybe wanted to piggy back on what sopra was asking and and sort of like your response there I was curious about how you think about um sort of building on top of other libraries with the ecosystem not just for find I saw there's a lot of like really interesting code around evaluations that that you have as well as quantization and so I was curious about sort of like if you could talk us through what that uh like development process looks like if a new paper comes out and it's really exciting because like a lot of what you thought talked about is exactly like how we think about touch T TCH and some of the libraries we've been developing like hackability over everything else so I'd love to sort of learn more about like what does that development process look like yeah uh that's a great question um so I think like often uh it it can like it can like depend a bit I guess um I think like usually ideally if we want to integrate like let's say like let's say like there's a new preference learning method or something um ideally we want to sort of like usually in the past we've taken something like one of our so for meth some of the methods it's very easy to sort of add so especially with these like dpos methods because it's kind of just like like uh a slightly different data processing pipeline because you're dealing with preference data now um and then a slightly different loss function and so often I'll sort of start with our existing like fine-tuning loop our existing training Loop and then look at like the open source uh code base of in this case I think in in dp's case I literally just went to like looked at the DPO implementation um and like I'm very happy to I think we're very happy to sort of like copy stuff in when it makes sense for evaluations this is sort of what we try to do is um we like it's it it's preferable if we can to uh sort of like copy in the sort of core logic of what's going on and then sort of add the sort of Integrations and other bits on top of that um and you know give credit where where possible where you know give credit where we've like you know copied particular bits of code over and so on um in some cases like some evals have like pretty nice packages that we can use so like our Pak EV Val has like a pretty nice package that makes it very easy to just put into sort of like a inference script but in other cases like I think for uh math or truthful QA um we've sort of like leaned more towards like going to the original implementation copying some stuff over and like thinking about some of the defaults and whether they make sense for us as well um and yeah really really I think a lot of it is like the designed structures ideally try to be uh closer to like uh yeah sort of like a collection of scripts uh with shared utilities where it sort of makes sense um but are really sort of designed to be more or less self-contained so that you know if something new does come along um it's very easy to either take some code that's already been out there for that method or you know take one of our existing scripts and change the core bits that that make sense um like those are the sort of two basic methods approaches um and then you know commit sort commit it up in sort of a new script uh yeah um yeah thanks heish for the talk it was very interesting uh I think one question I have is like given that you work with this kind of whole pipeline of like the whole posttraining world right like fine tuning alignment uh eail quantization like do you find that there are certain areas that are like not really that well served right now or where you have like more friction than you would like to have and you have to hack around stuff more than you would in an ideal world oh yeah that's a really good question um I feel like I would probably say like distributed training is always feels more more painful than it has to be um like it would be really nice if we could always just scale like if we could just hit a scale button and use like go like jump straight from like one or two gpus to multi Noe and we've to be fair it is pretty like it's not too bad right now for us with this repo but there's often the that ends up being sort of weird like often it takes like a bit more time than um we necessarily like uh the other thing I feel like is that uh sometimes it can be a bit annoying to sort of swap between so I mentioned before that like often times like generation is like a big bottleneck and say online training and swapping between um like often what you want to do is something like okay when I generate I want to like load up a VM engine generate a ton of like online examp but then I want to swap back to my regular sort of py for back Loop uh for uh then like doing some you know taking some po steps on that and this often involves like you know you have to be like okay I'm going to put like uh my inference is always going to sit on like this GP like you have to do some manual GPU memory management you have to sort of deal with the fact that sometimes you're you are dealing with like not really conflicting but you know you're dealing with like multiple systems of managing your models um and this can sometimes mean that you have to hack around things and um it's very much not not super duper seamless I feel like yeah I also observed that you have sort of uh I don't say complex but like you do think a lot about like data access patterns and just like mixing different data sets I'm assuming there's some online sampling involved and so on and like combining data sets how easy or hard has that part of this stack been like we've gotten a t ton of feedback on just like data set abstractions and especially as we move towards more modalities it's starting to become like a real challenge I was curious like if you could talk a little bit about that part of the pipeline yeah um I think like data is honestly like data mixing is is like a really honestly it's always like kind of feels pretty reasonable and straightforward we haven't done too much multimodality stuff quite yet um at least in open instruct um and this is probably something we'll move a bit to the start looking at a bit in the future um but I I feel like often the these days it feels like the like firstly the community has sort of like settled on some pretty reasonable ways of like storing your data and it's pretty easy usually to write some quick data trans like transform text Data into particular forms if you need to um and so honestly I think honestly if anything we spent a lot of time earlier this year trying out a lot of like fancy preference learning algorithms and like really trying to like dig into say like Po and what are the best ways to try and get it to work um but then going back and just being like okay what new data sets have come out um how do we can we just like add them to our mix do some quick ablations gave us like instantly way more bang for our buck uh they tried like a a trillion new sort of preference learning methods did uh in terms of like uh improving performance on our on a lot of the downstream evaluations that we cared about um so I feel like honestly data mixing is it's not it's it's not too bad right now I imagine with multimodal stuff it'll get more complicated um but a really big thing that maybe I think there's a lot of interesting research questions and also maybe some infrastructure questions is is sort of data curation is working out you know okay you know I can I can go and download everything on hugging face but downloading everything on hugging face isn't going to be uh going to give me a good model right I need to sit you can do a sort of like basic like grad student approach or like a bunch of hire or a bunch of research edas approach of like okay everyone go train a bunch of models on these data sets and we'll report our results and we'll look at them and then we'll decide on a mix and this works very well it's a very good Baseline but I think there's uh lots of interesting future work and lots of ongoing work as well going on in sort of like how do you identify what high quality data points are ahead of time how do you um filter out sort of noisy data how do you perhaps even just generate your own data ahead of time um to like sorry not ahead of time like generate regenerate your own data if like a new Mo a new better model comes out and you want to sort of take advantage of that in some way um and so I think like the basics of data mixed stuff seem like pretty set at this point but I think there's a lot in the sort of Realm of data curation and and working out data valuation SL attribution that's really interesting all right it looks like we have a couple of questions on the live stream um I'm just going to read it here uh the first question is is this something like instruct lab from Red Hat and IBM research uh I'm not like 100% familiar with this truck lab I think like there's a lot of uh stuff out there that is pretty similar um like I mentioned like like Tor trun has a bunch of a bunch of great scripts for like training various models likewise there's TRL and from hugging face and axel uh which I know is quite popular in the open source Community as well um I think like if it is struct we started it like some like a year and a half ago is uh when a lot of these like repositories were kind of still getting started to sub-degree um so we're sort of like developing it in concept with a lot of these things and it's more of a um it sort of serves like uh it serves research projects and then also like the sort ofo effort at ai2 first and sort of other things second um which might be different to other stuff out there um but yeah so it's I from a quick glance it seems like Vaguely Familiar and I think there there are a lot of like Frameworks out there that overlap I think ultimately you know it makes sense to just work with what you know right um and like what you can build easily on top of um like all these discussions about hackability before what matters a lot more is being able to have an idea implement it and run and test it as quickly as possible um as opposed to nice obstructions and so on cool should I just ask a question do you want to read it out um sure you can go ahead and answer it uh yeah data mixing and model merging uh and like the trade-offs between them are really interesting um so we found for example like I I I talked about this model earlier where we we trained this math specific model and then merged it back in in order to improve our math performance we actually did this partly because we found that there was negative interference occurring between uh that particular math data set that we knew was was really good really valuable uh the Numa data set and the other data we were using um so the uh so what we ended up doing like we ended up trying model merging and it worked like surprisingly well we didn't have you know we didn't have these negative interference issues and on average the sort of model improved um and I so and so what I've sort of like thinky that I I I have some like internal projects that are also sort of starting to point at this is that there are cases where just sort of mixing your data sets together might result in some sort of interference sometimes they're in different format it might be because they're in different formats it might be because um maybe you haven't balanced stuff quite well and doing and instead of like trying to perfect your mix instead just like training separate models and then merging them back back together can sometimes just be a much simpler and much more effective way of controlling that sort of like doing that sort of like capabilities mixing rather than trying to uh create this big one single data set that you're using um I also note that like sometimes model merging can be a good way to sort of uh test out uh how like particular capability or like test out sort of model ablations and get a good final model without sort of having to uh do a sort of big training run on one finalized mix you could sort of train you can sort of train a bunch of models in a bunch of different data sets check how they go pick the best ones and then merge them and you don't have to do any additional training um and I think like this is of going on in like the open source Community to some degree right of like people trying out different merges and so on um so I think it's a really interesting question there's like uh perhaps like interesting like unsolved questions about when when it might be better to merge versus when it might be better to uh combine data sets and we're still sort of working that out to some extent um I guess I I have one more question for you so you talk about like oh actually let's let's the I already got my turn let's let this one this one um I feel like this is maybe better for the P torch fix to answer um um yeah so uh I think in short it would be useful to learn P if you want to work on machine learning and uh deep learning um I'm not quite sure because the question seems a little incomplete but uh yeah feel free to check out P we have a lot of like uh documentation resources available online and you can always like reach out to us VI are GitHub if you have specific questions yeah EV go ahead um yeah my question was so you talk a lot about like ease of experimentation and that's one way that you can kind of like in yeah increase the rate that you can get new models out there like run your training Loop but like the other way that we can increase the rate of experimentation is just make the fine tuning Loops run faster like how important is that to you in your day-to-day experience yeah this is also a good question um I think like often the scale at like the scale of these data sets um and the scale of the like training runs that we're doing is often quite smaller than say like pre-training level runs and often like say for like a lot of the data sets that we have if we want to train like a AP model and like something like a million um instruction samples it takes like one or two days on the current sort of Hardware that we have um I think on like one node of like a I can't remember if it's A1 100s or h100s that we got that number off but anyway um and so like I I I feel like it's always it's always get it it's always good to get your fine tuning Loop faster for sure um and I'll happily take free like fine tuning uh sort of wins but at the same time I think often like often the sort of like the actual sort of tokens per second like the size of the stuff that we're dealing with is not quite large enough that like you know a really big reduction in tokens per second would make as much of a difference as doing some of the other sort of quality of life type stuff um if I cut down my like fine tuning loop from 12 hours to 6 hours that's great but to some degree I think being able to sort of oneclick run uh 15 different evaluations and have that sort of run and then like reported in one easy place is actually kind of more valuable than that uh like double speed up in trading time because uh running a bunch of those evaluations yourself and and building out that sort of sort of stuff uh and doing that a lot of that stuff manually would just be uh probably more annoying and uh use up more of your time and what a lot of people on the team end up doing is they'll sort of kickoff they might have you know six or seven different uh uh you know data sets or sweeps that they want to run and they can sort of kick those off and they'll finish overnight and then they can sort of come in in the morning and all the evals are sort of sitting there and so on and we can discuss whether or not it's worth add you know what what the sort of next action step from those experiments will be um I think also like generally speaking we also sort of like want to when it comes to things like evals and models there's always sort of new ones coming out and we always want to be able to add those and sometimes like overly engineering how well overly is maybe a strong word but like focusing a lot on Like Making Stuff faster can sometimes lock you in into particular choices that might uh harm you later down the line when it comes to trying to integrate other code bases or integrate other uh setups that you might care about aw any other questions yeah oh sorry go ahead I had a quick question on so you mentioned data set curation and understanding what data sets are high value I'm curious about if you know how well explored this area is are you guys using tools in the community to help you understand this or is this really just a manual process for you yeah uh originally it was all very like uh for for for a good while it was it has been mainly pretty manual in the sense of like running ablations and looking at the results and making decisions based off that um more recently we've tried to integrate some like more useful things uh and we like this is all very much work in progress so we' found for example that I mentioned this before that doing things like uh regeneration with existing models out there has proven to be like a really good way to sort of uh it firstly improves the quality of your data but also turns out these models pick up cases like oh this question is weird or like this question is malformed um and you can sort of pick out weird data points in your existing data sets um doing uh likewise doing sort of rejection like it turns out in our rejection sampling pipelines we also sometimes surface this where uh reward models like we'll find some data points that have like sort of outlier scores given to them by reward models and this usually points to some curation um but we're still sort of I think uh like working on building sort of more dedicated methods and tools for uh either doing like a sort of filtering type thing or doing a sort of more nuanced data mixing type thing where you know you might say like okay here's the evals I care about here's the data sets I have available how do I um construct like a really good mix based of this and i' I'd probably say that this seem this is like currently a sort of a very I think active research Direction the sense of you see lots of current work on like how to work out good pre-training data likewise I think there's like um a good space for like how to work out what really good instruction data um how to like mine really good instruction data from existing data sets out there so I think it's like there are some basic approaches that we've we've done right now I think it's mostly manual and I think the manual Baseline is also really hard to like automate a way to some degree right like it's very hard to beat sort of uh sitting down and just looking at results and thinking about them and then uh going on from there uh yeah okay um so I'm going to I I think the answer to that one is unknown at this point and uh we probably it's not really relevant to the topic of discussion so uh yeah maybe if you have a question about it might be worth using the discussion forums are creating an issue all right um any other questions from folks in the room or comments cool um yeah I think that concludes today's session thanks a lot Amish uh thanks for your time again and this was really insightful for us to learn as spy Dodge developers and I guess as users as well on how uh some of the research in the space is evolving um all right cool thanks everyone and I gu we'll see you all at the next one thanks thanks for having me