How Important are Benchmarks in Deep Learning?

Key Takeaways

And I see that you've done a whole bunch of benchmarking. Can you tell us a bit about like what motivated that work? I'm not going to lie and say like I love making benchmarks. I don't think making benchmarks is fun. Everyone wants to do the flashy thing like on train 100B model and then do cool stuff. For context to benchmarks that you're as referring to is multi-bench multimodal and kernel bench which is LM generated GPU kernels and evaluating those. And the most recent one is video game bench which evaluate vision language models that can play in a assortment of games. These are games from the 90s like Doom, Mario, Kirby. I think kind of highlights that we don't have a lot of good benchmarks. And the reason I ended up working on benchmarks every single time was because I wanted to work on a particular problem, but there was just no eval for it. I think this is actually is a is a really big problem in in the field in general. I think that like we kind of have a problem of evals currently aren't great indicator of how good a model actually is even on the task that you're evalling. So for example, there's a lot of evals for math and coding, but they're not great evals for whether or not a model is good at math or coding.