CS231n Winter 2016: Lecture 14: Videos and Unsupervised Learning

Key Takeaways

terms of administrative items um everyone should be done with sment 3 now if you're not done I think you're late and you're in trouble uh Milestone grades will be out very soon we're still getting through them and they're basically I think done but we have to double check a few things and we'll send them out okay so in terms of reminding you where we are in the class uh last class we looked very briefly at segmentation we looked at some soft detention models soft detention models are a way for uh selectively paying attention to different parts of the image as you're processing it with something like a recurrent neural network so it allows you to Selec pay attention to some parts of the scene and enhance those features and we also talked about spatial Transformer which is this very nice way of basically in a differentiable way cropping parts of an image or of some features uh either in an apine region or in any kind of warped shape or anything like that uh so a very interesting kind of uh piece that you can slot into your neural network architectures so today we'll talk about uh videos specifically now in image classification you should be familiar by now with the basic com that setup you have an image that comes in and we're processing it to for example classified in the case of videos we won't have just a single image but we'll have multiple frames so if this is an image of 32x 32 we'll actually have an entire video frames so 32x 32x T where T is some time extent okay so before I dive into how we approach these problems with convolutional neural networks I'd like to talk very briefly about how we used to address them before comets came about using feature-based methods so some of the most uh popular features right before convolutional networks kind of became very uh popular were these dense trajectory features developed by hang Wang at all and I just like to give you a brief Taste of exactly how these features worked um because it's kind of interesting and they Inspire some of the later developments in terms of how convolutional networks actually operate over videos so in Den trajectories what we do is we have this video that's playing and we're going to be detecting these key points that are good to track in a video and then we're going to be tracking them and you end up with all these little tracklets that we actually track across the video and then lots of features about those tracklets and about the surrounding features get accumulated in histograms so just to give you an idea about how this worked there's basically three steps roughly uh we detect feature points at different scales in the image I'll tell you briefly about how that's done in a bit then we're going to track those features over time using optical flow methods Optical flow methods I'll explain very briefly they basically give you a motion field from one frame to another and they tell you how the scene moved from one frame to next to next frame and then we're going to extract a whole bunch of features but importantly we're not just going to extract those features at fixed uh positions in the image but we're actually going to be extracting these features in the local coordinate system of every single track lit and so these histogram of gradients histogram of flows and MBH features we're going to be extracting them in the coordinate system of a tracklet and so hog here uh we saw histograms of gradients in two dimensional images there are basically um generalizations of that to uh videos and so that's the kind of things that people use to encode little spatial temporal volumes so in terms of the keyo detection part um there's been quite a lot of work on exactly how to detect good features and videos to track and intuitively you don't want to track regions in the video that are too smooth because you can't lock onto any visual feature and so there are ways for basically getting a set of points that are easy to track in a video so there are some papers on this so you detect a bunch of features like this uh then you run Optical flow algorithms uh on these uh videos so in Optical FL algorithm will take a frame and a second frame and it will solve for a motion field a displacement uh vector at every single position into where it traveled or how the frame moved and so here are some examples of optical flow results um basically here every single Pixel is colored by a direction in which that part of the image is currently moving in the video so for example this girl has is all yellow meaning that she's probably translating uh horizontally or something like that the two most common methods for using optical flow or for computing it uh at least maybe one of the most common ones here is Brock from Brock and Malik that's the one that is kind of like a default thing to use so if you are Computing Optical flow in your own project I would encourage you to use uh this large displacement Optical flow um method so using this Optical flow we have all these key points using optical flow we know also how they move and so we end up tracking these little tracklets of maybe roughly 15 frames at a time so we end up with these half a second roughly tracklets through the video and then we encode uh regions around these tracklets with all these descriptors and then we need to accumulate all these visual features into histograms and people used to play with with different kinds of like how do you exactly chunk at video spatially because we're going to have a histogram an independent histogram in every one of these bins and then we're going to basically create all these histograms per bin with all these visual features and all of this then goes into an svm and that was kind of the rough layout in terms of how people address these problems in the past question uh how do you def a track yes so track just think of it as um is going to be 15 frames and it's just XY positions um so a 15 XY coordinates that's a tracklet and then we extract uh features in the local coordinate system now um in terms of how we actually approach these problems with convolutional neural networks so you may recall Alex net on the very first layer we receive an image that is for example 227 X2 27 by3 and we're processing it with 96 filters that are 11 by1 applied at stri four and so we saw that with an Alex net this results in a 55x 55 by 96 volume um in which we actually have all these responses of all the filters at every single spatial position so now what would be a reasonable approach if you wanted to generalize a convolutional network into a case where you don't just have a 227 x 227 x 3 but you maybe have 15 frames that you'd like to encode so you have an entire block of 227 x 227 x 3X 15 that's coming into a convolutional network and you're trying to encode both the spatial and temporal patterns in inside this little block of volume so it would be like one idea for how to change a convolutional network or generalize it to this case good this is what we were doing like initially for our project so we down sample the each of the image and uh create a 4x4 grid and place the images side by side okay that's interesting so you're saying that you you basically took four frames and you arranged them as like blocks right like four cross 4 to 16 frames that's interesting okay I would expect that to not work very very well so the problem with that is that's kind of interesting basically all these neurons they're looking at only a single frame and then by the end of the conet you end up with neurons that are looking at larger and larger regions in the original image So eventually these neurons would see all of your input but they would not be able to very easily relate the L little spacial temporal patch in this image so I'm not sure actually if that's a real good idea okay go ahead did you like tie l M into this or something bottom could you tie an lsdm into it I think so we'll get to some methods that uh do something like that go ahead you could just stack in just like similar to how you do RGB your computional kernel okay so that's maybe a good idea so maybe you would take um so right now we have three channels and maybe you will take these frames and you'll actually just stack them across channels so you'll end up with 45 channels effectively and you could put a comet on that uh so that's something that I'll get to I think in the end you could do that I don't think it's the best idea as all when you want to share the parameters across for each convolutional filter across different time frames so that because things in one time frame are probably similar terms things in another time frame uh yes so you're saying that things in one slice of this time are you want to extract similar kinds of features in one time then a different time similar to motivation of doing it uh sharing weight spatially because features here are useful down there as well so you have the same kind of property where you'd like to share weights in time not only in space okay so building on top of that idea the basic thing that people usually do when they want to apply convolutional networks on videos is they extend these filters not only so you don't only have filters in space but you also have these filters and you extend them small amounts in time so before we have 11 by1 filters so now we're just going to have 11 x 11 by T filters where T is some small temporal extent so say for example we can use a two up to 15 in this particular case so for example if T was three then we have 11 by 11 by3 filters and then by three because we have RGB and so basically these filters are now you're thinking of sliding filters not only in space and carving out an entire activation map but you're actually sliding filters not only in space but also in time and they have a small finite temporal extent in time and you end up carving out an entire activation volume okay so you are introducing this time Dimension into all your kernels and into all your volumes they just have an additional time Dimension along which we performing the convolutions uh so that's usually how people extract uh the features and then you get this proper property where say t is uh three here and so then when we do the spatial temporal convolution we end up with this parameter sharing scheme going in time as well as you mentioned um so basically we just extend all the filters in time and then we do convolutions not only in space but also in time we end up with activation volumes instead of activation Maps Okay so some of these approaches were proposed um quite early on uh for example one of the earlier ones for in a context of activity recognition is maybe from 2010 so the idea here was that this is just a convolutional network but instead of getting a single input of 60 x 40 pixels we are getting in fact seven frames of 60 x40 and then their convolutions are 3D convolutions as we refer to them so these filters for example might be 7 by seven but now by three as well and so you end up with a 3D con and um these 3D convolutions are applied at every single stage here um similar paper also from 2011 with the same idea we have a block of frames coming in and you process them with 3D convolutions so you have threedimensional convolutional filters at every single point in this convolutional network so this is in 2011 uh very similar idea also so these are from before actually Alex Nets these approaches are kind of like smaller neural network and convolutional network so the first kind of large scale application of this uh was from this awesome paper in 2014 by Carpathia tal so this is for processing videos uh um so the model here on the very right that we uh we called slow Fusion that is the same idea that I presented so far these are three-dimensional convolutions happening in both space and time and so that's slow Fusion as we refer to it because you're slowly fusing this temporal information just as before we were slowly fusing the spatial information now there are other ways that you could also wire up uh convolutional networks and just to give you some context historically this is um I was in Google research and Alex Nets just came out and everyone was super excited because they work extremely well on images and I was in the video analysis team at Google and we wanted to run uh comets on YouTube videos and but it was not quite clear exactly how to generalize uh you know Comal networks on images to videos and so we explored several kinds of architectures of how you could actually wire this up so flow slow Fusion is a 3D com kind of approach early Fusion is um this idea that someone described uh earlier where you take a chunk of frames and you just concatenate them along channels so you might end up with a 227 by 227 by like 45 well everything is just stacked up and you do a single comve over it so it's kind of like your filters on the very first con layer have a large temporal extent but from then on everything else is a two-dimensional convolution so in fact we call it early Fusion because it fused the temporal information very early on in the very first layer and from then on everything is just 2D con you can imagine architectures like l convolution so here the idea is we take two Alex Nets we place them say 10 frames apart so they both compute independently on these 10 10 frames apart and then we merged them much later in fully connected layers and then we had a single flame Baseline that is only looking at a single frame of a video so you can play with exactly how you wire up these uh models um for the slow Fusion model you can imagine that if you have threedimensional uh kernels now on the first layer you can actually visualize them and these are the kinds of features you end up learning on videos so these are basically the features that we're familiar with except they're moving because now these filters are also extended a small amount in time so you have these little moving blobs and some of them are static and some of them are moving and they're basically detecting motion on the very first layer and um so you end up with nice moving volumes go ahead how much uh uh how deep in time do we go for the fail is it does it still apply that we should try to make it as small as possible yeah so your question is how much do we extend this in temporal length for the filters we'll get to that and I think the answer is probably yes just as in spatial um it works better if you have smaller filters and you have more depth it the same applies I think in time and we'll see an architecture that does that in a bit go ahead what exct is the out of the full comp like what what Vector are you expecting at the very end I see so in these cases um oops in these cases we're expecting the same we're just classifying every single so we have a video and we're still classifying fixed number of categories at every single frame but now your your prediction is not only a function of that single frame but also a small number of frames a lot a lot on both sides so maybe your prediction is actually a function of say 15 frames half a second of video so we end up with fun uh moving filters in this paper we also released a video data set of 1 million videos in 500 classes just to give you context for why this is actually it's kind of difficult to work with videos in um right now I think because problem right now I think is that there's not too many very large scale data sets like on the order of millions of very varied images that you see in imet there are no really good equivalents of that in the video domain and so we tried with the sports data set back in 2013 but I don't think it actually we fully achieved that and I think we're still not seeing very good really large scale data sets and videos and that's partly why we're also slightly discouraging some of you from working on this on projects uh because you can't pre-train these very powerful features because the data sets are just not quite there um and other kind of interesting things that you see and this is why we also sometimes caution people from uh working on videos and getting very elaborate very quickly with them is because sometimes people they think they have videos and they get very excited they want to do 3D convolutions lstms and they just think about all the possibilities that open up before them but actually it turns out that single frame methods are a very strong Baseline and I would always encourage you to run that first so don't worry about the motion in your video and just try single frame networks first so for example in this paper we found that a single frame Baseline was about 59.3% uh classification accuracy on our data set and then we tried our best to actually take into account small local motion but we ended up only bumping that by about 1.6% so all this extra work all the extra compute and then you ended up with relatively small gains and I'm going to try to tell you why that might be but basically video is not always as useful as you might intuitively think and so here are some examples of um kind of predictions that we uh these are different data sets of sports and our predictions and I think this kind of highlights slightly why adding video might not be as helpful in some settings so in particular here if you're trying to distinguish Sports and think about trying to distinguish say tennis from swimming or something like that it turns out that you actually don't need very fine local motion information if you're trying to distinguish tennis from swimming right lots of blue stuff lots of red stuff like the images actually have a huge amount of information and so you're putting in a lot of additional parameters and trying to go after these local motions but in most in most classes actually these local motions are not very important they're only important if you have very fine grained categories where the small motion actually really matters a lot and so a lot of you if you have videos You'll Be inclined to use a spaer temporal crazy video networks but think very hard about is that local motion extremely important in your setting because if it isn't you might end up with results like this where you put in a lot of work and it might not work extremely well let's look at some other uh video classification uh convolutional networks so this is B from 2015 it's relatively popular it's called c3d and the idea here was basically V network has this very nice architecture where it's 3x3 comp and 2 x two pool throughout so the idea here is that basically let's do the exact same thing but extend everything in time so going back to your point you want very small filters so this is everything is 3x3x3 com 2x2 X2 to pool throughout the architecture so it's a very simple kind of vet in 3D uh kind of approach and that works reasonably well and you can look at this paper for reference uh another form of approaches actually that works quite well is from Karen simonian in 2014 Karen simonian by the way is of vggnet Fame uh he's a person who came up with the vget he also has a very nice paper on video classification and the idea here is that he didn't want to do three dimensional convolutions because it's kind of painful to have it implemented or find it and so on so he only used two dimensional utions but the idea here is that we have two comets looking at an image and the other one is looking at Optical flow of the video so both of these are just images but the optical flow basically tells you how things are moving in the in uh the image and so both of these are just kind of like an vgg net like or Alex net like Comet one of them on optical flow one of them on the image and you extract Optical flow with say the Brocks method that I've shown you before and then you let you fuse that information very l in the end so both of these come up with some idea about what they are seeing in terms of the classes in the video and then you fuse them and there are different ways of fusing them and so they found for example that if you just use a spatial Comet so it's only looking at images you get some performance if you use a comet on just the optical flow it actually performs even slightly better than just looking at the raw images so Optical flow actually here in this case contains a lot of information and then if you fuse them you actually end up performing even better now an interesting point to make here by the way is is that if you have this kind of architecture especially here complex this 3x3x3 filters you might imagine that actually would think that I mean why does it help to actually put in Optical flow you'd imagine that in this end to end framework we're hoping that these componets learn everything from scratch in particular they should be able to learn something that simulates the computation of computing Optical flow and it turns out that that might not be the case because sometimes when you compare uh video uh networks on only the raw pixels and then you add Optical flow it works better and so I think the reason for that is um probably comes back to actually data since we don't have enough data we have small amount of data I think you actually probably don't have enough data to actually learn very good Optical flow like features and so uh that would be my particular answer why actually hard coding Optical flow into the network is probably helping out in many cases so if you guys are working on your projects with videos I would encourage you to actually try maybe this kind of architecture where you extract Optical flow and then you pretend that it's an image and you put comets on it and you merge it and so that seems like a relatively reasonable okay so so far we've only talked about little local information in time right so we have these little piece of like movement say of like half a second and we're trying to take advantage of it to do better classification but what happens if you have videos that actually have much longer temporal um temporal kind of dependencies that you'd like to model so it's not only that the local motion is important but actually there are some events throughout the video that are much larger in time scale than your network and they actually matter so event two happening after event one could be very indicative of some class and you'd like to actually model that with a with a neural network so what are the kinds of approaches that you might think for trying to actually you know how would you modify the architecture to actually model these kinds of much longer term events go ahead attention model okay so attention model perhaps so You' maybe like to have an attention over you're trying to classify this entire chunk of video and maybe you'd like to have attention over different parts of the video yeah idea go ahead sometimes you look at images very different scales we could do the same thing in in time or we just skip frames um I see yeah that's a good idea so you're saying that we have these multiscale approaches where we process images on very low detail level but also sometimes we resize the images and process them on the global level and so maybe the frames we can actually like speed up the video and put a com that on that I don't think that's very common but it's a sensible idea I think yeah it's good so the problem roughly is that basically this temporal in extent is maybe sometimes too short it doesn't span entire many seconds so how do we make architectures that are a function of much longer time scales in their prediction good could you include a hidden stur okay yeah so one idea here is we have this video and we have different classes that we' like to predict at every single point in time but we want that prediction to be a function not only of a little like chunk of .5 seconds but actually a much longer time extend and so the idea that is sensible is you actually use recurrent neural networks somewhere in the architecture because recurrent networks allow you to have infinite context in principle over everything that has happened before you up till that time so actually if you go back to this paper that I've already shown you in 2011 it turns out that they have an entire section where they CH take this and they actually have an lstm that does exactly that so this is a paper from 2011 using 3D com and lstms way before they will cool in 2011 uh and so this paper basically has it all they model little local motion with 3D com and they mo model Global motion with lstms and so they put lstms on the say the fully connected layers so they string together fully connected layers with this recurrence and then when you're predicting classes at every single frame you have infinite context and so this paper is uh is I think quite ahead of its time and it basically has it all except it's only cited 65 times I'm not sure why it's not more popular I think people I don't know but basically this is a way ahead of it time paper that recognized both of these and actually used lstms way before I even knew about them so since then there are several more recent papers that actually kind of take this similar approach so in 2015 by Jeff Donahue atal from Berkeley the idea here is that you have video and you'd like to again classify every single frame but they have these comets that look at individual frames but then they have also lsdm that string this together temporally a similar idea also from a paper from I think this is Google and so the idea here is that they have Optical flow and images both are processed by comets and then again you have an lstm that merges that over time so again this this combination of uh local and Global so so far we've looked at kind of two architectural patterns in accomplishing video classification that actually takes into account temporal information um modeling local motion which for example we extend 3D com where use Optical flow or more Global motion where we have lsdm that string together sequences of mning time steps or fusions of the two now actually I'd like to make the point that there's uh another cleaner very nice interesting idea that I saw in the some paper and uh that I like much more and so here's basically the rough picture of what things look like right now we have some video and we have a 3D Comet say that is using optical flow maybe or it's using 3D Com or both um on a chunk of frame chunk of your data and then you have RNs on the top on the fully connected layer or something like that that are doing the long-term modeling and so kind of what's kind of not very nice or unsettling about this is that there's kind of this ugly asymmetry about these components so you have these part these neurons inside the 3D Comet are only a function of some small local chunk of video and then you have these neurons on the very top that are a function of everything in the video uh because they're recurrent units that are a function of everything that has become come before it and so it's kind of like an unsettling asymmetry or something like that so there's a paper that has a very clever and neat idea from a few weeks ago that is much more nice and homogeneous in a vget like style uh where everything is very nice and homogeneous and simple and so I don't know if uh anyone can think of how we could what we could do to to make everything much more cleaner and I couldn't because I didn't come up with this idea but I thought it was cool once I read it go ahead maybe you could reverse and do the RNN before the interesting you want to do the RNN before the comet actually starts processing the images uh I'm not sure what that would give you so you would have RNs on the raw pixel information and then come that's on top of it somehow uh you would certainly have neurons that are a function of everything um but it's not clear what the lstm would be doing in that case like would it be blurring the pixels it's too low level probably processing at that point good so it seems like one of the problems is that in video there's so much duplicate information between two frames that like it there's a lot of redundancy if you're only looking at a few frames so you could have maybe like an ensemble of Confidant that looks at different like temporal resolution so this conent is looking every frame right but you could have another Convent that's looking like every two frames every four frames every eight frames and you can feed all of those into rnet I see uh so your idea is that I think it's similar to what someone pointed out where you take this video and you work on multiple scales of that video so you speed up the video and you slow down the video and then you have 3D comets on different um temporal like uh speeds or something like that I think it's a sensible idea yeah don't you just do background subtraction and get only the stuff that's interesting to look at could you do background subtraction and only look at things that are interesting to look at I think that's a reasonable idea I think it kind of goes against this idea of having end to end learning because you're introducing like this explicit computation that you think is useful simplifying learning basically I see yeah good could we have weight sharing between the 3D compet and the RN can we have weight sharing between the 3D compet and the RNN um that's interesting I'm not 100% sure because the RNN is just a hidden State vector and you're doing Matrix multiplies and things like that but in the com layers we have this like spatial structure so I'm not actually sure how the sharing would work um but yeah maybe okay so the idea is that we're going to uh let's see we're going to get rid of the RNN we're going to basically take a comet and we're going to make every single neuron in that Comet be a small recurrent neural network like every single neuron becomes recurrent in the comet okay so the way this will work and I think this is beautiful uh but their picture is kind of uh kind of ugly so I'm not sure if this makes makes sense so let me try to explain this in a slightly different way what we'll do instead is so in a normal com that we have a con layer somewhere in the neural network and it takes input from below the output of a previous com layer or something like that and we're doing convolutions over this to compute the output at this layer right so the idea here is we're going to make every single convolutional layer a kind of a recurrent layer and so the way we do that is we just as before we take the input from below us and we do comms over it but we also take our previous output from the previous time time step of this Con layer's output so that's this Con layer from previous time step in addition to the current input at this time step and we do convolutions over both this one and that one and then we kind of have um you know we've done C and we have these activations from uh current input and we have activations from our previous output and we add them up or something like that we do a recurrent like recurr Network like uh merge of those two to produce our output and so we're a function of the current input but we're also a function of our previous activations if that makes sense um and so what's very nice about this is that we're in fact only using two dimensional convolutions here there's no 3D com anywhere because both of these are width by height by depth right so the previous comp volume is just with height depth from the previous layer and we are with height depth from previous time and so both of these are two-dimensional convolutions but we end up with kind of like recurrent process um in here and so one way to see this also with recurrent neural networks which we've looked at is that you have this recurrence where you're trying to compute your hidden State and it's a function of your previous hidden State and the current input X and so we looked at many different ways of actually wiring up that recurrence so there's a VIN RNN or an lstm or there's a GRU which Gru is a simpler version of an lstm if you recall but it almost always has similar performance to an lstm so Gru has slightly different update formulas for actually performing that recurrence and so what they do in this paper is basically they take the gru because it's a simpler version of an lstm that works almost just as well but instead of every single Matrix multiply it's kind of like replaced with a con if you can um if you can imagine that so every single Matrix multiply here just becomes a conve so we convolve over our input and we convolve of our output and that's the before and the below and then we combine them with the recurrence just as in the gru to actually get our activations and so before it looked like this and now it basically just looks like that so uh we don't have some parts infinite in extent and some parts finite we just have this RNN conet where every single layer is recurrent it's Computing what it did before but also it's a function of its previous outputs and so this RN and com net is a function of everything um and it's very kind of uniform it's kind of like a vggg net you just do 3x3 com 2 X2 Max pool and you're recurrent and that's it maybe that's just the answer it's the nicest simplest thing so in summary whoops where's my summary um so if you'd like to use spaer temporal conval networks in your project and you're very excited because you have videos the first thing you should do is stop and you should think about whether or not you really need to process local motion or Global motion or if motion is really important to your classification task if you really think motion is important to you then think about whether or not you need to model local motions are those important or are the global motions very important based on that you get a hint of what you should try but you always have to compare that to a single frame Baseline I would say and then you should try using optical flow because it seems that if you especially have smaller amount of data it actually uh is very important it's like a very nice signal to actually encode that and explicitly specify that Optical flow is a useful feature to look at and you can try this gr RCN Network that I've shown you just now but I think this is too recent and experimental so I'm actually not sure if I can fully endorse this or if it works it seems like it's a very nice idea but it hasn't kind of been proven yet and so that's uh that's kind of like the rough layout of how people process videos in the field so I don't know if there's any questions because Justin is going to come next go ahead I'm wondering if architectures like the G RCN you last mentioned have been used for NLP uh architecture like Gru RCN so this one uh has it been used for uh NLP um that's a good question I don't think so I'm not super duper expert on NLP but I haven't seen this idea before so I would I would guess that I haven't seen yeah I'm not sure I don't think so good I have an answer for that there's a model really similar to this but it's more similar to the see in the bottom on the top it's from y Yim who is working at har with Sasha right now so does it look more like this then uh yes yeah okay yeah okay question do people ever do like networks that take into account the audio also do people take into account networks that also take in audio and somehow consider it in on a side with a video I would say that that's definitely something uh people would want to do um um you don't see too many papers that do both of them just because people like to kind of isolate these problems and Tackle them um maybe not jointly but certainly if you're in a company or trying to get something working in a real system you would do something like that uh but I don't think that there's anything that you would do you'd probably do this with a late Fusion approach where you have a whatever works best on videos whatever works best on audio and then you merge that somewhere later somehow but that's certainly something you can do yeah it's a good idea and with and with the neural networks right it's very simple because you just have a fully connected layer that's looking at the output of both at some point and then you're classifying as a function of both yeah all right cool so we're going to go into un surprise larning and I guess we have to yeah you have to switch Yeah so it's our first tag team lecture hopefully it works all right can you hear me is that is that hooked up okay okay so uh I guess for the rest of the lecture we're going to switch gears completely and entirely and talk about unsupervised learning um so I'd like to make a little bit of a contrast here that first uh we're going to talk about some sort of basic definitions around unsupervised learning and then we're going to talk about uh two different sort of ways that unsupervised learning has recently been attacked by Deep learning people um so in particular we're going to talk about autoencoders and then this idea of adversarial networks and I guess I need my clicker right right so pretty much everything we've seen in this class so far is um supervised learning so the basic setup behind pretty much all supervised learning problems is that we assume that our data set has sort of each data point has sort of two distinct Parts um we have our data X and then we have some label or output y that we want to produce from that from that input and our the whole goal in supervised learning is to learn some function that takes in our input X and then produces this output or label Y and if you really think about it um pretty much almost everything we've seen in this class is some instance of this supervised learning setup so for something like image classification um X is an image and then Y is a label for something like object detection um X is an image and then Y is maybe a set of objects in the image that you want to find um y could be a caption and then we look at captioning um X could be a video and now y could be either a label or a caption or pretty much anything anything so I just want to make the point that supervised learning is this very very power powerful and generic framework that encom encompasses uh everything we've done in the class so far um and the other point is that supervised learning actually makes system that work systems that work really well in practice and is very useful for practical applications um unsupervised learning I think is a little bit more of an open research question at this point in time so it's really cool um I think it's really important for solving AI in general but at this point it's maybe a little bit more of a research focused type of of area um it's also a little bit less well defined so if in unsupervised learning we generally assume that we have just data we only have X we don't have any Y and the the the goal of unsupervised learning is to do something with that data X and the something that we're trying to do really depends on the problem so some so in general we hope that we can discover some type of latent structure in the data X without explicitly knowing anything about the labels so some classical examples that you might have seen in previous machine learning classes would be things like clustering so something like K means where X is just a bunch of points and we discover structure by uh classifying them into clusters um uh some other classical examples of unsupervised learning would be something like principal component analysis where X is just this uh some punch of data and we want to discover some low-dimensional representation of that input data so unsupervised learning is this really sort of cool area but a little bit more problem specific and a little bit less well- defined than supervised learning so two things that two uh architectures in particular that people in deep learning have done for unsupervised learning are these ideas uh is this idea of an autoencoder um we'll talk about sort of traditional autoencoders that have a very very long history we'll also talk about variational autoencoders which are this sort of new cool Bean Twist on them we'll also talk about some generative adversarial networks that actually are this really nice idea that let you generate images and model uh sample from natural images so the idea with an auto encoder is is pretty simple um we have our inputs X which is some data and we're going to pass this input data through some kind of encoder Network to to produce some features uh some latent features Z so this you could think this stage uh you could think of a little bit like a learnable principal component analysis where we're going to take our input data and then convert it into some other feature representation so those uh many times these X's will be images like these Carr 10 images shown here um so this this encoder Network could be something very complicated so for something like PCA it's just a simple linear transform but in general this might be a fully connected Network um originally uh sort of maybe 5 or 10 years ago this was often a single layer fully connected network with sigmoid units now it's often a deep uh deep network with ra units and this could also be something like a convolutional network uh right so we also have this idea that uh Z the the features that we are that we learn are usually smaller in size than x so we want Z to be some kind of useful features about the data X so we um we don't want the network to just uh transform the transform the data into some useless representation we want to force it to actually crush the data down and summarize its statistics in some useful way that could hopefully be useful for some Downstream processing but um the problem is that we don't really have any explicit labels to to use for this Downstream processing so instead we need to invent some kind of a surrogate task that we can use using just the just the data itself so the the surrogate task that we often use for autoencoders is this idea of reconstruction so since we don't have any y's to learn a mapping instead we're just going to try to reproduce the data X from those features Z and especially if those features Z are smaller in size then hopefully that'll Force the network to act to summarize the to summarize the useful statistics of the input data um and hopefully discover some useful features that could be um one useful for reconstruction but more generally maybe those features might be useful for some other tasks uh if we later get some supervised data so uh again this decoder Network could be pretty complicated uh when Auto encoders first came about often times these were just simple either a simple linear Network or a small sigmoid network but now they can be deep Ru networks and often times these will be um I I decided UPC convolutional is a good term so it's nice and small and fits on the slides so often times this decoder nowadays will be one of these UPC convolutional networks that takes your features that are again are smaller in size than your input data and kind of blows it back up in size to reproduce your your original data um and I'd like to make the point that these things are actually pretty easy to train so on the right here is like a quick example that I just cooked up in torch so this is a four layer encoder which is a a a convolutional network and then a four- layer decoder which is an UPC convolutional Network and you can see that it's actually learned to reconstruct the data pretty well um another thing that you'll sometimes see is that these encoder and decoder networks will sometimes share weights um with as just sort of as a regularization strategy and with this intuition that these are opposite operations so maybe it might make sense to try to use the same weights for both so just as a concrete example um if you're in if you think about a fully connected Network then maybe your input data has some Dimension D and then your latent data Z will have some smaller Dimension H and if this encoder was just a fully connected Network then the weight would just be this Matrix of D byh And now when we want to do the decoding and try to reconstruct the original data then that we're mapping back from H back to D so uh we can just reuse the same weights in these two layers if we just take a transpose of the Matrix so when we're training this thing we need some kind of a loss function that we can use to compare our our reconstructed data with our original data um and then once uh and often times we'll see L2 a simple like L2 ukian loss to to train this thing so once we've chosen our encoder Network and once we've chosen our decoder Network and chosen a loss function then we can train this thing just like any other normal n Network where we get some data we pass it through to encode it we pass it through to decode it we compute our loss we back propagate and everything's good so once we train this thing then often times we'll take this decoder Network that we spent so much time learning and we'll just throw it away um which seems kind of weird but the reason is that reconstruction on its own is not such a useful task so instead we want to apply these networks to some kind of actually useful task which is probably a supervised learning task so here the setup is that we've learned this encoder Network which hopefully from all this unsupervised data has learned to has learned to compress the data and extract some useful features and then we're going to use this encoder Network to initialize part of a larger supervised Network and now if we actually do have access to maybe some smaller data set that has some labels then hopefully this most of the work here could have been done by this unsupervised training at the beginning and then we can just use that to initialize this this bigger Network and then fine-tune the whole thing with hopefully a very small amount of supervised data so this is kind of the dream of UNS one of the dreams of unsupervised feature learning that you have this really really large data set of with no labels you can just go on Google and download images forever and it's really easy to get a lot of images um the problem is that labels are expensive to collect so you'd want some system that could take advantage of both a large huge amount of unsupervised data um and also just a small amount of supervised data so autoencoders are at least one thing that has been proposed that has this nice property but in practice I think it tends not to work too well which is a little bit unfortunate because it's such a beautiful idea um another thing that I should point out almost a bit of as a side note that if you go back and read the literature on these things um from maybe the mid-2000s uh in the last 10 years then people had this funny thing called greedy lir wise pre-training that they used for training Auto encoders and here the idea was that um at the time in 2006 training very deep networks was was challenging and uh if you you can find quotes in papers like this that say that even when you have maybe four or five hidden layers it was extremely challenging for people in those days to train deep networks um so to get around that problem they instead had this uh Paradigm where they would try to train just one layer at a time and they use this this thing that I don't want to get too much into called a restricted boltman machine which is a type of graphical model and they would use these restricted boltzman machines to kind of train these two these little layers one at a time so first we would have our our input image of maybe size of size W1 um and this would be maybe something like PCA or some some other kind of fixed transform and then we would hopefully learn um using a restricted boltzman machine some kind of relationship between those first layer features and some higher layer features um when once we once we learn this layer we would then free freeze it and learn another restricted boltman machine on top of those features connecting it to the next layer of features um so by using this type of approach it let them train just one layer at a time in this sort of greedy way and that let them hopefully find a really good initialization for this larger Network so after this uh greedy pre-training stage they would stick the whole thing together into this giant Auto encoder and then find tune the auto encoder jointly so um nowadays we don't really need to do this um with things like Ru and proper initialization and batch normalization and slightly fancy fancier optimizers this type of thing is not really necessary anymore um so as an example um on the previous slide we saw this four layer convolutional deconvolutional autoencoder that I trained on cifar and this is just fine to do um using all these modern neural network techniques you don't have to mess around with this uh greedy layerwise training so this is not something that really gets done anymore but I thought we should at least mention it since you'll probably encounter this idea if you read back in the literature about these things so the basic idea around autoencoders is I think pretty simple it's this beautiful idea where we can just use a lot of unsupervised data to hopefully learn some nice features um unfortunately that doesn't work but that's okay um but there's maybe some other nice type of task we would want to do with unsupervised data uh question first sorry can you just explain how exactly you train the w um so you take your data and what that outut and yeah so the question is what what's going on here um right so this is this is um this is maybe you could think about a three- layer neural network so our input is going to be the same as the output so we're just hoping that this is a neural network that will learn the identity function but um that's a really and in order to learn the identity function we have some loss function at the end um something