Pytorch Quick Tip: Weight Initialization

Key Takeaways

[Music] welcome back for another pie torch video I thought I would just make a quick video on how to initialize weights for a network my torch has inbuilt initialization which works quite well normally so you wouldn't have to worry about it but if you want to know how to change it that's what we're going to learn in this video so first of all there are a bunch of different initialization techniques like Xavier initialization or timing initialization and you can read more about them in the Python documentation I'll link to it in the description but for now let's just get started on how to initialize our network so what I have here is a very simple CNN example really nothing special here just a comm a pool and another column and then the linear layer so what we're gonna do is we're gonna define another function to find initialize weight which just takes self and then what we're gonna do is we're gonna do for EM in self dot modules and this is tracked by by PI torch so when we define all of these modules here they come the max polka and linear etc these are all stored in the self taught modules so we're just gonna go through each module one by one so this would be the entire network essentially if we just do print em and we run the function here like this we would get so back the entire comm one where's the pool yet pool comm to fully connected yeah so we get all of them in this case so what we're gonna do is we're gonna we're gonna say if this instance m2 the current module if it's a comm 2d we're gonna do so you could use a very variety of different initialization techniques we're just gonna do the climbing uniform coming uniform and then we're gonna do MDOT wait so what this means we're just gonna perform climbing uniform on the the weight of that specific module and we're only gonna do if if it's a comm 2d and then also we're gonna do so you can define a bias in the convolution it's default the default is true so the you know it initializes a bias by default but we can do if MDOT bias is not none we can do and then dot in it constant and advice and we're just gonna set the bias to zero yeah but you could do other things as well you could do now we don't have a Bachelor in our network but let's say we would have okay and dot NN that batch norm 2d in dentists yep so we're gonna do is we're gonna do ending that in a dot constant m dot wait comma 1 and this is just standard initialization for bathroom and then the by I mean the yeah the bias and the bias should be zero and you could also do something like else it is instance M comma and and dot linear and then this then we would for example we could do the same thing here as we did for the and then come to D you could also do different things maybe you want to have normally distributed values or something like that or you want to use Xavier whatever and then in and out in it constant m dot pius we'll just set it to zero so it's just an example of how you initialize the weights and then you would called initialize wait after you defined all of your modules that's it hopefully this was a this video was useful thank you so much for watching the video and hope to see you in the next one