Maximizing Data Quality & Model Efficiency

Key Takeaways

is basically maximize the the relevance of the the raw information, meaning like maximize the the the quality and the diversity of the data. This is probably like the most important one. It's not really the the one that people are often the the most excited about, but I think it's pretty well known now that like there is this garbage in, garbage out uh thing that it's basically super important to have very good data and like a very good diversity. But you can also optimize for other thing and this is quite orthogonal to the data set. So you need to have a team that build a very good data set. You also need to have a team that do research on model architecture to basically have the like uh construct the best model possible under certain constraints and the constraint are like the the efficiency. So you want a model uh that will be uh good it at inference uh that will like fit on a certain number GPU that will have a certain like KD cache uh consumption and like that. So this is like the the the model of static charts and when you are yeah you have clicks like this uh data set and model arch you can also maximize the like the information you get from the data. So at each step you basically want to maximize like how much you extract from the data. Uh one like uh way to do that is to do disloc distillation.