Rephrasing Low Quality Data Boosts Performance

Key Takeaways

This is like the curve between lowquality web and high quality web. So this is the web basically. And if you only rephrase the low quality parts, you get this curve, the light blue one, which is kind of close to to the one where you also rephrase. I know it's not that, but it's like it gives you a big boost. And I think I'm not sure if they do the ablation here, but I'm not. Yeah. Okay. So that's what they do. they they also try to to rephrase like the high quality subsets and they also repeat the data set they rephrase on and there is only a small difference but you could think it's because of the repeat I'm not even sure it's because of that to be honest and the the the the thing that you you need to understand from it is like it's very good to refra the low quality data to have like just better better benchmark and better performance and and yeah and rephrasing is is a bit uh is a bit tricky to do. Uh I think uh a lot of it is making basically the right prompt >> and one of the the good prompt I mean the good style that the the model learn uh the model like especially because the benchmark file constructed this way is the conversional uh data uh for example MMLU so this is from small M2 uh uh training so which is uh 1.7B I think >> yeah I know >> I don't think very small. >> Yeah, very small. And this is basically until like 6.5 trillion of token. So, which is a lot to be honest. until this amount of token the MMLU in the QA format meaning that the the model have to select uh which answer is uh the model have to output