Structured pruning with Weights & Biases | Model optimization made simple

Key Takeaways

And then something that is near and dear to my heart because I'm um applying pruning methods right now in my work is pruning. And so the idea here is to take the lower magnitude weights and either zero them out or even just remove them completely from the model. And there's structured and unstructured techniques. With unstructured, you allow yourself to kind of remove them without any imposed constraint on the structure that you're moving. So you could imagine having like a weight matrix and being able to remove a weight at an arbitrary location. But one challenge when you're trying to deploy to the hardware is that um you need some mechanism to keep track of all of the remaining nonzero weights. And that's something that's right now more challenging to do given the hardware constraints that we are working with. Um, so I've been working with structured pruning where you're removing entire rows or columns of weight matrices or channels and um then in the end you fine-tune uh to restore the performance and have a smaller model to deploy. Um, but I think something I want everyone to take away is that this is a very iterative process as you've all mentioned. Sometimes we're trying like one or more of these methods at once and it results in a lot of different versions and experiments and we really want to have a good way to keep track of our learnings and the different experiments that were done and that's kind of where Weights and Biases has come and