Applied Deep Learning 2023 - Lecture 9 - Preprocessing, Augmentation, Regularization, Visualization

[Music] you might not need shoes to run but Dam it they can help welcome back to another installment of the applied deep learning course today we're going to take a look at pre-processing data augmentation regularization and a bit of visualization and see how we can strap some shoes on our Network so they run faster and better but before we do that let me quickly recap from last week's lecture last week we talked about Transformer so what is a Transformer a Transformer basically is an encoder decoder neuron Network that we can use for sequence processing in a similar way how we used recurrent neural networks with one benefit that a Transformer does not not have any recurrency in it so we um don't have this bottleneck in performance where we need sequential processing but we can feed in the entire input sequence at once the most essential building block that enables us to perform this operation is an attention mechanism that we have in our in uh in our Network how this works is that the uh attention mechanism learns how individual words of our input relate to other words or elements from our input relate to other elements and can learn how to focus and how things relate to each other if you just take the entire sequence in one um as once into your input why do we need positional encoding so especially for um natural language processing uh we need to encode somehow the information where where a word appears in our input it could be at the beginning it could be at the end somewhere in the middle and depending on where it appears the word might mean something differently the solution is to kind of also attach some sort of um positional information to give the network the chance to also understand wait it's a same word but it appeared somewhere differently so um it can make some sense out of this what is the difference between the first um attention Block in the encoder and the first um attention Block in the decoder well there are several differences and I think the most important one is that the uh first attention Block in the decoder is a so-called masked attention um and why is that a masked attention because it gets at the output sequence and it has to learn how words relate in our output sequence to each other but if we feed in the entire output sequence and it should protect uh predict the output sequence well doesn't make doesn't make much sense right because it would just learn to predict whatever next word it's reading in so in order to solve this we're masking out all the words that we haven't pred predicted yet that are kind of from the future and the network has to learn to predict them and for the next step we can then use those words and if we fit in everything at once we need to kind of mask out all values in on the right top of our Matrix to remove them from our input so the network is forced to um look at some other parts of the input for knowledge and it's not allowed to see any of the output of Target that it should predict in the future how can Transformers be used for object detection I would say there are at least three things that can help in that regard and that were also described in the paper um Transformer detectors and first of all you need some sort of feature extraction so instead of using word to Vector we basically use a standard convolutional neuron Network to to extract features from our Network then we feed it to our an attention mechanism that learns how um different parts of an image relate to each other the second part is that for the decoder we're having so-called object queries and those object queries are being transformed so one object query will be transformed in one output and what those object queries are are um specific inputs kind of learned guesses um that say okay take a look at the left top and tell me if there is a kind of a small or a large object something like that you can think of it in that way and those object queries are learned so you have say 100 object queries and then you have a maximum of 100 output predictions and each object query learns where to look and for kind of what of shape of object they should uh look out for and the third part is once we have our output prediction we need to um Define a loss and relate our output prediction with the ground truth set of boxes and the me mechanism they proposed uh in the paper was bip partite matching loss where you find try to find the globally optimal assignment from the prediction to the output to the ground truth and you do this by um comparing the boxes and the class labels and using the Hungarian algorithm to find um an the optimal assignment between those two sets in polinomial time so that's the 5% matching loss in the Nal and if you put all this together you can actually perform object detection with Transformers now to the shoes that help us running why do we need something like pre-processing so there are a couple of reasons why pre-processing can be um necessary or at least beneficial whenever you train a neuron Network um there are many parts of your network that assume normalized data so input normalization is one major factor that you should definitely consider uh for example some default parameters like the learning rate assume that your data is normalized also the parameter initialization strategy Xavier initialization or he initialization they also assume that your data is somehow normalized so it can kind of make sense how to initialize um initialize all those weights in your network and of course every parameter update step um is ring on this assumption basically so when you think about the gradient um what it does is it's we are Computing the partial derivatives of all our parameters with respect to one single parameter to learn how to update that parameter but the underlying assumption therefore is that we only change one parameter at a time and all other parameters stay the same which is not what we're doing what we're actually doing is we're updating all parameters at the same time so so uh when we're working on unnormalized data this can degrade our performance significantly because the gradient can then easily explode or vanish as the distribution that we have going from one layer to the next layer kind of constantly changes so if you have normalized uh data in your network layers uh you can kind of overcome this problem so the basic approach for doing this would be that you compute some statistics on your training set and then you apply the very same operation on the training set the validation set and of course your test set input normalization as I just said is one of the most common and most important um input operations that you should definitely have in your network it's a very common pre-processing step and it helps us to um get gain invariance to some undesired variations per sample so if you have an image that is a bit brighter or a bit darker that slight variation shouldn't affect whatever we're trying to achieve so we can gain some invariance to these um variations intra class variations and ultimately it can support our training if we normalize our entire training set set to make it look more homogeneous the underlying assumption is the following that certain properties of our data are relevant and other properties are not and the network should learn to distinguish between things that are relevant and that are not relevant uh for example if you try to understand what color uh what type of fruit you you're depicting in an image the color is definitely very important and can help you you to distinguish say an apple from a lemon but the brightness of the image or the contrast should not affect it the shape yeah the rotation of the object no so there are some properties where we can try to kind of normalize our input uh to make it easier for our Network to learn okay those things are always the same like they all have some same kind of um brightness distribution so I can basically safely ignore that otherwise we might run into the situation with the ruler if you remember the example from the very first few lectures where the network learned to um take a look at the ruler instead of taking a look at the melanoma to classify whether it's good or B so um another example would be the audio sampling rate if you work in the audio domain think about it if you uh sample your input with completely different rates it and then just feed those to the network it might look as if someone was talking much faster or slower which is not at all related to the content that is being recorded in that audio and finally if you rotate a page with text it shouldn't def like definitely should not uh change the content that is written on that page we can try to normalize our input data to kind of even those slight variations out uh and try to come up with a very homogeneous training set so here you see an example of people who are not normalized and once we normalize our input for the brightness and the contrast they all uh look at least to me uh much more homogeneous image normalization let's talk about this how you can actually implement this what I've just showed you uh brightness normalization can be done by by mean substraction so you just compute the mean brightness of all your samples and then substracted contrast normalization by a division of the standard deviation or you can do histogram Equalization to take a look at the um histogram and try to even that out and for column what you're typically doing is you're doing it per channel so here are some examples of your original and then you apply some sort of normalization or histogram Equalization and you can see that the images now are much more looking like the others pre-processing can even be essential to make your network work at all so if you present this page page of handwritten text to a machine and ask it so please tell me what's written on that page uh I mean even for humans this is very hard but a machine will probably just trug and say uh I have no idea how should I proceed here so applying the right pre-processing step makes this problem go from unsolvable with neuron networks to very hard and challenging which is already a big step forward one way how you can actually do this for text it's a very common procedure that you try to find the baselines of your written text try to find the line where text is being written upon and um this Baseline detection is now a pre-processing step that you can use and that transforms the problem in a kind of one-dimensional problem where you just follow along the Baseline and then you can feed that part of the input into your Nal network of course the image is still a two dimensional um image snippet but by onedimensional I mean that you can now only read from left to right or right to left so you just have to follow that Baseline instead of trying to navigate where in the image you should read your next letter pre-processing is definitely domain specific so if you're working on text you might need different pre-processing than when you're working on music scores what I've been doing on music scores was also domain specific so kind of the basine if you will for music scores are those five lines and those are called stuff and you try to extract one stuff at a time uh to break down the problem and so what you would do is kind of try to find those stuffs and then process them in individually coming from making data look more similar we can also now try to make data look more different the idea um is that ideally we want to train on as much data as we possibly can with a large set of variations that capture the underlying distribution fully and as we um add more data to our date training set we usually increase our performance because the data now reflects our problem much better we can kind of approximate the underlying distribution to a higher degree unfortunately getting more data is not always possible or it might actually involve human suffering if you think about um data sets that contain pictures of of people suffering from certain diseases or or rare diseases it can be very difficult to get more samples of them so ideally we want to kind of get around and work with whatever data we have with as much State as we can possibly gather but sometimes we have to do with a little bit less data augmentation therefore comes handy where we try to create meaningful fake data what data augmentation does is that we're trying to use transformations to increase our training set size which is what exactly what we want while at the same time staying within our distribution and not changing the semantic of our input data obviously the effects of Performing data augmentation can now be observed as the following uh we can increase the diversity in our data set if you think of an image that is only being recorded at a perfect angle and then you start rotating it our network will learn how to cope with images that are slightly rotated because it has seen them them now during training if all images are perfectly aligned and then at test time when you're performing inference you show it a slanted image it will not work that well because it's never seen that in the input which brings me exactly to the second point point this slight variations we're now gaining some some form of robustness against those slight Variations by applying the right augmentation methods it improves generalizibility to samples that we haven't seen before and on the downside it also increases our computational costs because we now have to pre-process our images somehow to look a bit differently each time we take that that image and train on it this is mostly done online so we perform our data augmentation as we are performing our training As and we're not even storing those images for example when you let's go back to the image domain um when you're performing an image augmentation like a rotation we can take an a sample loaded from disk or if it's already memory even better rotate it and then feed it through our Network for the training step and the next time we're loading that image we can apply a different rotation and if you would store all the images with all possible uh Transformations and augmentations this would not fit anywhere uh on your disk that's why you typically do this online and you're not storing those images or those objects but just before performing them online now if you perform this online it has the drawback that you're not only um using your GPU for the training but you also quite heavy on your CPU because most of the time those Transformations are being performed on the CPU so as a practical tip if you perform heavy data augmentation make sure that you also have a powerful CPU and enough RAM on your computer to enable this and not have the data augmentation be the bottleneck of your training procedure finally coming back to this um semantic of the data be careful with whatever kind of transformation you apply some Transformations do not change the semantic others do if you flip an image uh horizontally it might be okay for a cat but in other word uh terms when you are working with text A B then could look like a d or if you perform vertical mirroring well you would get something that now says something completely different so make sure that whatever transformation is you apply as statea augmentation does not alter the semantics in your particular data set the most common Transformations for images are rotation random uh scaling where we kind of try to zoom in or zoom out of an image uh random cropping where we just C cut off some parts and um of course mirroring at least for cats this works fine because a cat that you mirror horizontally is still a cat you can even apply some of the more advanced augmentation techniques for images which could be sharing some kind of nonlinear stuff where you or some cutouts where you just drop certain parts of your input image uh masking similar idea and uh even some really fancy three-dimensional Transformations that make your image look as if you kind of had a physical print out of that photo you crumbled it together and then unfolded it again which uh gives you something that looks like this I think this is a fantastic idea um there are some great tools I've linked one tool here in our references that you can go and check out to see if any of those U Transformations might be useful for you another example for data augmentation that should not alter the semantics is so-called adversarial training let's start with an image you're on a plane and you're in the cockpit and the cockpit has hundreds of knobs and turns that you can change so changing one knob slightly shouldn't do much however if you change all the knobs even just for a tiny little bit at the same time the effects can be quite dramatically that's the idea of adversarial training that even neuron networks that have human level performance or nearly um like performance that exceptionally well all have a nearly 100% error rate on carefully crafted adversarial examples so those are malicious example that try to break your network an adversarial example is more formally defined as X Prime that is near X so the inputs are very similar but y Prime is quite different from y so the output is completely different different and the reason why this is happening is because most neuron networks are built primarily out of linear building blocks and those linear building blocks are very sensitive to simultaneous changes in numerous inputs changing just one neuron usually doesn't have such a big impact but if you change all neurons even just a tiny little bit these linear building blocks are very sensitive and can uh have severe effects on the output if we perform adversarial training we would enforce locally constant behavior and how we do this is basically by just adding a little bit of random noise to our input we really have to be careful with our neuron networks if you put them out in the wild in the future because like we are the ones who might be building the next object detector or something that powers self-driving cars however if it's so easy to fool these networks we should really be careful on what we rely on and how we can maybe try to make those networks robust against these advisal examples um self-driving cars is just one example but think about say fraud detection by having a neuron Network that tries to detect fraud um and kind of breaking into this and understanding what you need to alter to break the network and to say well there is a positive you can carefully craft an attack uh against your neuron Network and this idea of trying to attack a neuron Network by adding a little bit of noise to break it has become its own research field because the more systems that we actually deploy and use and rely upon the more important it is to protect ourselves against such attacks in some extreme cases the authors of this paper have shown that even just changing one single Pixel can be enough to alter the predictions of the neuron Network they employ the special method to find out how to change the input like which pixels are are having the biggest effect and here in those images you can see that they highlighted the single Pixel that they actually changed with a red circle but I think it's very impressive that there is a method that can change your input so little like just a single Pixel and still have an effect on your Network's output I really love that example in this paper they show an an adversarial attack on modern object detectors with convolutional neuron networks and what they did was they carefully crafted an image that does not produce any output in our object detectors and if you print that pattern on a hoodie and you wear this hoodie you can actually show that you become invisible to these object detectors so this is kind of a modern day invisib ility cloak if you do not want to be tracked by uh one of those modern object detectors that we've talked about in the past when we now leave the domain of images we can go into the audio domain and also here there are a bunch of data augmentations that you can apply to your data for example you can inject noise you can shift your samples in time you can change the pitch to make it sound a little bit higher or lower or you can just change the tempo of what you're you've recorded and ideally if I say a sentence slowly or if I say it very fast it shouldn't make a difference in whether you understand what I'm saying or not there is a great library that enables you to do many of these augmentations it's called lip Rosa go check it out if you're working with audio let's take a uh quick look at some examples of these augmentations how they sound like so that's the original let's shift the end to the beginning so it cut some part of the end and put it to the beginning or it was maybe a different part of of the of something that came before we can add some [Music] noise or we could stretch it in time maybe you didn't hear it that well let's play it again so you could hear it was much slower than the first few ones or at least a bit slower than the first few ones furthermore you can perform many of these data augmentation operations for audio actually in the image domain the idea is that for audio processing many modern approaches rely on spectrograms where the audio waveform is being transformed into an actual image and when you transform it into an image you all of a sudden have access to all of the operations that work well on images for your audio domain you can perform time War Waring uh where you kind of randomly increase the speed or decrease the speed in your input you can perform time masking which you can see over here where you we just cut out a small part of our input like these two parts or we can do frequency masking where we remove an entire frequency spectrum from our input so one sample just does not have any information say the frequency range between 10 and 12 KZ then that will be frequency masking if we go to the text domain you can also apply some meaningful data augmentation techniques here like synonym replacement where you replace one word with another word that has the same meaning a synonym you can perform random insertions where you just attach another random word somewhere in your input and the network should still kind of be able to make some sense out of it you can swap words randomly or you can delete Words randomly of course you might break the syntax because as said Superior human out on the road of life well actually that one does make sense but maybe with random swap you can actually break your syntax Asad Superior human comedy played out on roads back the life the of Life yeah this doesn't definitely does not adhere to Standard English syntax so that input sample would be broken however we randomly perform these operations and the network should see the correct order also often enough you can even make use of some kind of errors that are being introduced when you translate sentences into another language the idea would be that you perform back translation you take an input you translate it to another language and translate it back and what you will get back is a slight variation of your input because words can have many meanings and it's not necessarily that you end up with the same input that you fed into the original translation so this is kind of a clever mechanism how you can um increase your Text corpus to have more samples by just translating it and then translating it back if your translation works fine it should keep most of the syntax and semantic intact and this and your Corpus should still have the same meaning data augmentation is also a domain specific in the same way that pre-processing is domain specific if you want to perform data augmentation on images of Music scores you can create new music uh you just randomly generate a sequence of notes and then you render it and you get a new piece of music it doesn't have to sound good it just has to look like real scores if you can achieve that you can generate as many samples as you want as part of your data augmentation you just create music scores on the Fly and then you can train your network to perform operations on these just remember when you're relying on synthetic data the process by which you generate those samples will determine your underlying distribution therefore you will always need some real data in your data set as well because the variation that you will get from those synthetic samples might not cover the entire true distribution that you could encounter so just keep this in mind coming to the next topic where you now bind our shoelaces regularization the central problem in machine learning is how to make an algorithm perform well on previously unseen data we don't necessarily have the network per perform exceptionally well on the training data we want it to work good on the test set and when we perform inference on new unseen data regularization is one technique that can help us achieve this more efficiently regularization is a set of strategies that are designed to reduce the test errow potentially at the risk of ending up with a higher training error what we're doing is we're limiting the model capacity to avoid overfitting on the training set data augmentation as we've just seen before is one mechanism how you can perform some sort of regularization and it's actually a really efficient strategy but there are also a couple of other strategies that we're going to take a look at some add restrictions to our parameter values some add maybe an additional objective function we've already talked about these ex auxiliary functions for example in um reinforcement learning so we can encode some sort of prior knowledge that we have about the underlying problem and of course we can use ensembles where we have multiple networks that work together model assembles are as I just said modle networks that are trained independently but then are being used jointly the idea is that you have the the same model but you initialize it differently every model is initialized with random variables and then they're trained on the same data set if you start from a different point with a different initialization the idea is that you end up at a different Optimum so your network might make certain mistakes but another Network that was trained on the same data but started somewhere differently might make a different mistake and not make the mistake that the first Network made so if you run your sample through multiple networks and then kind of take a count a v voting process to see which one uh should be the desired output then you're kind of averaging your predictions and you're trying to avoid uh errors that only one of those networks produced you're kind of a quorum now that votes for which output to actually predict weight Decay is also a very popular regularization mechanism and it's adding an additional objective function by adding a parameter Norm penalty what we're doing is that our objective when we describe our parameters of our par uh when we describe our learning that instead of just Computing the gradient descent we also add a panalty term for how big our weights in our neural network are we typically only do this for the kernel for the uh multiplicative weights and not for the bias term and typically we also use the L2 U parameter Norm penalty that we can compute just by having a transposed weight multiplication by with the weights if we use weight Decay our gradient update step now becomes slightly different uh as you can see here we have the weights and then we subtract our uh gradient that we compute so what happens here is that the coefficients that we're Computing our weight our new weights will shrink for every update step that we perform that's why it's called weight K our weights become less and less and the rate by how fast they become zero ultimately is determined by our weight Decay parameter Alpha and that is typically a very small number between 0.00001 or 001 weight decay in practice for example with Caris is that you basically just specify in your layer kernel regular regularizer and you say well take the L2 Norm that's basically weight Decay uh for pych there is a specific parameter called weight Decay and you just specify that one parameter that says this Alpha value of how fast your weights Decay towards zero one of the most important regularization mechanism that I highly recommend every one of you to use is early stopping how it works is that you monitor your training on the validation set you should do this anyway uh by just saying okay after each Epoch compute some statistics some loss maybe some Precision or any meaningful value on your validation set and you save the models during training and then you perform from like after each Epoch you save the model and then you continue training the next Epoch so when you are at the end of your training instead of returning to the last Model that you trained the last training step of the last EPO you actually return to the epoch that had lowest loss on or the low the best performance on your validation set it could be that that moment was already a long time ago but what that means is that your network contined to training and overfit your training set so monitor your validation set and if the performance on your validation set does no longer improve for a certain time you can stop training allog together it's highly effective uh simple to implement and basically almost a free lunch so take this um and put it into your neuron networks whenever you train something with machine learning and neuron networks the effect of this early stopping is that we control the effective capacity by limiting the number of steps the optimizer can take weight Decay does a similar thing if our weights Decay towards zero we are limited by that amount of steps that we can take in our network if we take too many steps with a too high weight decay all of our weights will be just zero uh the good thing about early stopping when you compare it to weight Decay is that it automatically determines the correct amount of regularization so we don't have to worry whether we use the right Alpha value for this weight Decay because you just monitor on your validation set and stop when you don't see any further Improvement cutout is something that I think I mentioned before so you can take for example for images certain parts of your input and just remove them set them to to zero uh and kind of make the network uh forc to be able to understand images even though some parts of the images are hidden Dropout is a very common regularization strategy that I think lost a bit of popularity recently uh or actually over in the last few years however you can still find it in quite a few networks what it does is that you have your network and with a certain probability P you set the neuron's output to zero regardless of what you feed into that neuron just produces no output it's independent for each sample or for each mini batch what neurons are disabled for that pass so you're discarding neurons during training and when you perform inference on your final set or when you do testing on your test set all neurons are active so what is the effect of this you force the network to be to not rely on individual neurons too much because they could just drop out and then they wouldn't produce any output so the neuron network has to also learn to rely on different paths and different features of your input because one particular feature could just become zero at one point and network still has to produce a good output the effect of applying Dropout to your neuron network is pretty similar to an assemble so your subset of neurons that you have in your network uh works as if you had trained another Network and in the end you deactivate Dropout for inference and for testing so all of the networks that you trained in parallel are now working and operating at the same time batch normalization is a regularization strategy that is a bit controversial but very popular and you can see it in almost every neuron Network in in one point especially whenever we talk about convolutional neuron networks in general uh it is hard to train a large neuron Network because of so-called internal covariant shift what is this internal covariate shift uh we Define this internal coari shift as the change in the distribution of the network activations due to the change in network parameters during training so what happens is that in one layer here we perform a certain operation and that output follows now a certain distribution if we now train on a different sample that distribution that is being produced is different so even if your input data is normalized we have zero mean and unitary variance the distribution of these intermediate layers is constantly changing back and forth and the gradient tells us how to update each individual parameter but the underlying assumption is that the all of the other layers that we computed our gradient for does not change because we're just Computing the partial derivatives with respect to just one parameter so this constant shift in our Network can become a serious issue batch normalization initially came across this problem problem and thought well we can then normalize our layers so what you do is that you think okay let's try to combat this internal coari shift by calculating the mean and the variance of the input and then we normalize our layer inputs to calculate uh with some calculated statistics that we compute during training and fix during inference and then scale and shift our data set to obtain the output and how we should scale and shift our data is learned during training that's batch normalization interestingly batch normalization has seen quite some controversy uh because people thought that it was combating this internal coar shift so the Assumption was well we know that internal coage shift is bad we think that batch normalization is reducing that so we know that empirically when we train and we use batch normalization that the performance increases so well batch normalization therefore Works due to reducing this internal coari shift however there were some researchers that had doubts about this hypothesis and they did what every good researcher does they went and take took a look at whether this is true or not this hypothesis so they questioned this and they showed that it does not in fact reduce this internal covariant shift in some cases they could even show that it increased those internal coar shift then the question is what does batch normalization do and they found out that instead of reducing this internal coar shift it smoothens out the gradient and smoothening out the gradient has a similar effect of imp improving our performance so if that is our gradient we're trying to get the ball down to the lowest point we have these bumps and then we might have to take steps in the wrong direction or we get back up because we bounce up instead we have a much smoother gradient and it becomes much easier for the network to learn how to optimize on our training data set so the good news is batch normalization Works exceptionally well in practice and it's definitely advisable to use it uh even if some researchers say you should definitely not use it should use something else instead um so which is the bad news that it's not fully understood um why it's working so well I mean we know that it smoothens out the gradient but there's still some doubts on on how exactly batch normalization works batch normalization in practice can uh replace some other regularization techniques like Dropout uh and the question then is where should I put this bch normalization and there is no clear answer to this people have put it uh before the activation after the activation somewhere in between uh one of those modules and you can see that there was not a clear window of saying you should always put it behind the activation or before the convolutional layer or after the convolutional layer so there are some some I would say papers that explored this but I haven't come across any papers so far that was able to say you should always put the bachor mod station at this point meaning that you basically have to experiment and see how it changes your network and maybe putting it right after the convolutional layer is much better than putting it somewhere else one thing I would like to point out here is that bch normalization isn't the only way to stabilize the gradient layer normalization is a popular alternative to batch normalization and how it works is that it directly estimates the normalization statistics that we've seen before from the sumed inputs to the neurons within a hidden layer what that means is let me visualize this for you if we have a batch with three samples here and those are our input features X1 through X4 then what batch normalization would do is it would calculate the statistics U the mean and the variance for each feature so here across the features X1 2 3 and four um but as you can imagine this is now sensitive to the size of of our batches so if you have a small input batch size the output statistics that we compute here are very sensitive and you also have to imagine that for training and for inference we need a slightly different set of operations to perform uh batch normalization layer normalization on the other side would compute those statistical values for normalization um for each sample so not across feature but for an entire layer for each sample here and there are other normalization methods to for example instance normalization or group normalization one activation function that can actually also function as a regularization mechanism is the so-called scaled exponential linear unit celu and it has been described in a paper called self-normalizing neuron networks and the hypothesis is that when you use that kind of um activation function your neuron Network automatically normalizes its output to be stable and if this network output from each layer is stable and does not have this internal coage shift we don't need batch normalization uh and this is how you can compute this it's quite a simple to compute activation function and and I think it would be interesting to also try this out in a couple of neuron networks to see whether they actually perform better stochastic weight averaging is a simple yet very effective way to improve the performance of your Optimizer by averaging the weights what does it mean here to average the weights what we do here is we take two basic ingredients to improve the performance of our neuron Network and the underlying idea is that usually uh we reduce the learning rate of our Optimizer towards um like a very low value to converge towards a single solution that solution is ideally the globally optimal solution but if you visualize the gradient surface the solution so what we end up with typically is kind of a large flat area and the solutions that our optimizers do is because of the volume of this thing typically at the edge of this flat surface so how about we explore this flat surface a little bit and try to find the center of it so what stochastic weight averaging does is it basically says a let's modify the learning rate and keep exploring good Solutions so so you can see it in the left bottom image here is that um initially we start with a high learning rate and then we perform our usual decrease in learning rate for say 75% of the training and for the remaining 25% of the training what we do is we uh keep actually a learning rate that is not too low so we keep a moderate learning rate and we keep track of the weights that we compute here after a couple of steps this allows the network to you know keep exploring good Solutions and once we're done with the last 25% of our training we take those snapshots of those weights and we average them and compute the average values and as you can see here also for batch nalization this would require some special handling but I'm not going to get into those details what you can see on the right side however is very interesting you see the blue curve the test error and the green curve the training error or the training loss more precisely here and what's interesting is the discrepancy between the curves uh and the solutions that they provide so from a pure optimization performance stochastic gradient descent the green um thing here would give us the lowest value of the training loss but when you compare that to the test era that we obtain you see that it's not the optimal solution and they've shown that for many many um cases and in many scenarios this stochastic weight averaging that you perform like for the last 25% you memorize the weights and then you average them out allows you to um find the center of of this large flat surface that generalizes better and therefore obtains a lower test error and stochastic weight uh weight averaging Works in tandem with any other Optimizer because you just perform stochastic grade in descent as you would do it before but you just keep track of those weights and average them in the end so it's a very simple yet effective way how you can improve potentially the performance of your neuron Network even further coming to the last part of our lecture today is visualizing our training and visualizing our Network one thing I can absolutely recommend is the tzor board which gives great insights into your network how it performs and is an easy to use interface for exploring your data exploring your uh training Pro progress and monitoring your progress as you train your neuronetwork it's available for all sorts of platforms and Frameworks uh it was initially created for tensor flow but it can be used with many other deep learning Frameworks as well I'm now in Google colub again and this is the sample notebook that tensorflow provides to play around with uh the tenser board and the interesting thing is that the tenser board uh of obviously works Standalone but you can even embed it into a Google colop notebook and for demonstrating how this works is that we train a simple Network here to I guess perform image classification it's the yeah the standard thing that they're trying to do so when we're starting so we're training here for a bunch of Appo say for five Appo and then we can open the tensor board uh when we just point it to our locks and what we get is a beautiful interface that is white now here it is and we get a beautiful interface that uh we can use to explore some of our data so we train it for five app pox they were numerated from 0 to four here and you can see the accuracy for each EPO uh on your training set and our validation set as we move along here so that's really interesting and you see okay Network performed much better in the fourth EPO which was to expect and uh you can use also that if you have to to manually perform early stopping but just looking at those graphs and then stopping yeah I think they're also demonstrating this here that you can start the T supp boort and then run the training and it will dynamically update over time so you can have kind of a separate process that performs this tensor board that runs the tensor board and your training process and they run in parallel and it automatically loads data from the from the disk where the logs are being written to some of the interesting aspect of this tenser board is the vi wide variety of features that it offers uh one would be of course showing some scalers if we have an image processing problem we can also and images are being written to our tens board we can also take a look at those images for that example here no images are being written but we can explore the graph so that is our neural network we can also uh explore some of the distributions of our individual layers and maybe we can see that they look kind of okayish here they're centered around zero but it could be that some layers completely drift off or do some crazy stuff uh you could maybe see it here and some of the interesting diagrams here that I'm not fully uh aware of how to interpret them uh basically you're seeing some values here over time like for each EPO here how the distribution looked like I guess so feel free to play around with tensor board and find out what else it has to offer it's it's a great tool I think and I'm using it for basically what whenever I train neuron Network I'm automatically writing information to the T in the tenser board format to disk and then just run t board to quickly be able to play around with this and explore my data one very nice tool that I've come across to visualize your neuron Network architecture is called netron and I highly recommend that you check it out it supports a wide range of tools and you can drag in your configuration or your model and then visualize the architecture with all of its you know parameters Etc here um in a nice visual way another very powerful method of exploring your data and exploring your network is by so-called Class Activation Maps the idea is that we map the weights of our output layer back to our feature Maps so what we do is we take so we have an image uh and we feed it through our Network and that obviously results in certain features that are being computed before the final St state that performs in an operation so we take those features we multiply them with the weights of our last layer and that gives us a an activation map which says which features were related to which part of our last layer which still has the spatial information for images for example and what you can see here is one of these uh I think really fantastic visualizations of these Class Activation maps to be able to say okay the network learned to CL classify whether this person has disease or Not by looking at this area so it can kind of say doctor the network predicted this to be uh symptomatic um because it had something over here and then the doctor can use that information to look at that particular part of the input and be more efficient in trying to diagnose patients as I said before what we're doing is we're Computing uh the features in our output and then we're multiplying them with uh the previous layers we can do it with the last uh layer we can potentially also multiply the features that we have with earlier layers thinking back of um Class Activation maps in the context of global average pooling if you think back and you remember what global average pooling was it was the idea that we have we kind of shape our output in such a way that we have one layer per output class if we have some sort of image classification problem and then uh we just map that class to our output to not kind of reshape our input image but just perform global average pooling now if we use global average pooling the effect would be that the last layer is the feature map for individual um types of our output for individual classes so if we multiply this uh we can see how much it activated for certain parts uh for certain classes in our output and maybe uh even learn how close certain classes are to each other let's take a look at some code for these Class Activation Maps what we have here is our beloved mnist data set where we perform um image classification and then what we're trying to achieve is that we're going back to those Class Activation Maps so we take our usual data set we pre-process it to load it into memory uh so if we just show a random image we can see okay that's an eight of course those are black and white images but if you visualize it in terms of gray scales you can also uh show it as a sort of heat map and then we have a fairly simple neuron network with a few convolutional layers and global average pooling at the end which has 10 outputs so in that case it's not the global average pooling we still have a um so it has the 128 things uh but then a dense layer afterwards so we reduce it by using global average pooling which is another way of how you can do it and then we just train our model and after a couple of steps we get 99% accuracy that's a nice visualization of the model and what happens now is that we try to evaluate this so we run um our data set again or some samples from our test set and what we use is we first get the weights of the last layer before the end so those features I guess now from the global average pooling if we have say 128 anyway what we take here are the weights from those last layers from the global average pooling and when we're running it through our model we're producing multiple outputs one would be the standard classification a and the other thing here is that we actually want in our outputs now for this uh the the information from the layers before like near the end so the last few layers so minus 3 1 2 3 would be this convolutional layer I guess and minus one would be this dense layer if I'm not mistaken so we predict a model on our test set and then what we do is exactly what I said before we're multiplying here we're doing the dot product of the features that we extracted so those are like the features from one image that are zoomed to the size that we want to have with the weights from those um from those last layers that we have here so that I guess it's the the the dense layer here at the end and what we get is is a heat map that we can now overlay over our input uh to get these beautiful images which state okay the neuron Network mostly relied on the information here to say whether this is a two or not which I guess makes sense because most other numbers look quite differently here so if you have a one looks differently here three goes in the other direction four four five so all of the other digits if I'm not mistaken have um a different shape down here and even if you have a four it would even make more sense to look at this part because that particular curve here would have another line here 4 four so 4 One it makes sense to take a look at everything in the surrounding if you just have a straight one well there's probably nothing and so on and so forth so you can see the network trying to make some sense out of these inputs if it's directly interpretable or not um I leave it up to you but I think it it can be at least helpful for certain problems if we're able to see um if we're able to see which parts of the network were active and the network was looking at it's kind of the uh attention mechanism but for us humans I've attached one more notebook from the slides here where we're using um where we're not using Max pooling but I leave that for you to explore and see the effects of what happens if you have your network without Max pulling there is an idea of visualizing not just the activation Maps but also the training process in itself and the training process in itself is the gradient so we can try to understand how our network is kind of getting to the bottom of the valley the same idea that I try to give you with this gradient descent and there is a gradient visualization toolbox that you can try out and play around with and there you can see uh how the gradient looked like and Network tried to go down to the valley here and maybe find some interesting ideas I just have to warn you visualizing the gradient uh is feasible for small problems however when you have a very large neuron Network it can be computational expensive to compute that visualization suming up today's lecture I hope you got some ideas for how you can and fine-tune and improve your networks by using data normalization as it is a very important pre-processing step because many of our parts of our neuron Network like initialization algorithms rely on the Assumption of having normalized input and having normalized weights during training like bch normalization CeeLo can be mechanisms to enforce this pre-processing can make make some problems go from the domain of unsolvable to at least somehow solvable so they become tractable and as always you need domain specific knowledge for those pre-processing steps to be able to develop them and kind of have this initial idea of how you want to pre-process your data to make it work large networks need proper regularization I told you before the best performing neuron networks are those that have a huge capacity but are well regularized so what we can do is we can U employ several mechanisms at the same time like data augmentation where we create meaningful data we can uh employ weight Decay early stopping is something where we stop when we see no more improvements on our validation set which is both per beneficial for the output performance for the overall performance as well as we're saving time so I would definitely recommend this one uh Dropout where we can intrinsically train model oromos or batch normalization and finally when you visualize your gradient or your activation Maps it can help you to get a somewhat understanding of what your network is internally doing and where it's looking we explore this a bit more in the final lecture of this course where we talk about explainability and explainable AI but for now I leave you with that I wish you a wonderful week and I'll see you next week