CS231n Winter 2016: Lecture 12: Deep Learning libraries

Key Takeaways

today we're going to go over these four major software packages that people commonly use for de learning um as usual a couple administrative things um uh the the Milestones were actually due last week so hopefully you turn them in um and we'll try to take a look at those this week um also remember that assignment three the final assignment is going to be due on Wednesday so um any of you guys done already okay that's that's good um and you have late days so you should be fine um another another thing that I should point out is that if you're actually planning on using terminal for your projects which I think a lot of you are then make sure you you're backing up your code and data and things um off of the terminal instances every once in a while um we've had some problems where uh the instances will crash randomly and in most cases the the terminal folks have been able to get the data back um but it sometimes takes a couple days um and there's been a couple cases where um actually people lost data because it was just on Terminal and it crashed so I think um if you are planning to use terminal then make sure that you have some alternative backup strategy for your code and your data um so like I said today we're talking about uh these four software packages um that are commonly used for deep learning uh Cafe torch theano and tensor flow um and as a little bit of disclaimer at the beginning I've Mo like personally I've mostly worked with Cafe and torch so those are the ones that I know the most about um I'll do my best to give you a good flavor for the others as well but um just throwing that disclaimer out there uh so the first one is Cafe um we saw in the last lecture that really Cafe sprung out of this paper at Berkeley that was trying to reimplement alexnet and use alexnet features for other things um and since then Cafe has really grown into a really really popular widely used package for especially convolutional their networks so Cafe is uh from Berkeley um that I think a lot of you people have uh know and it's mostly written in C++ um and there's actually bindings for CAF so you can access the Nets and whatnot in Python and mat lab that are super useful um and in general Cafe is really widely used and it's really really good if you just want to train sort of standard feedforward vanilla convolutional networks um and actually Cafe is somewhat different than the others other Frameworks in this respect that you can actually train uh big powerful models in Cafe without writing any code yourself so for example the the resnet image classification model that won IM that won everything last year you can actually train a reset using Cafe without writing any code which is pretty amazing so the most but the most important tip when you're working with Cafe is that um the documentation is not is sometimes out of date and not always uh perfect so you need to not be afraid to just uh dive in there and read the source code yourself um it's it's C++ so hopefully you can read that and understand it um but in general the the C++ code that they have in Cafe is pretty well structured pretty well organized and pretty easy to understand so um if you have doubts about how things work in Cafe you your best bet is just to go on GitHub and read the source code um so the m so Cafe is this huge big project with like probably thousands tens of thousands of lines of code and it's a little bit scary to understand how everything fits together but there's really four major classes in Cafe that you need to know about um the first one is a blob so blobs um store all of your data and your weights and your activations in the network so these blobs um are things in the network so your weights are have blob your weights are stored in a blob your data which would be like your pixel values are stored in a blob um and your labels your y's are stored in a blob and also all of your intermediate activations will also be stored in blobs so blobs are these uh n-dimensional tensors sort of like you've seen in numpy except that they actually have four copies of an n-dimensional tensor inside um they have a data ver a data version of the tensor which is is um storing the actual raw data and they also have a parallel thing that a parallel tensor called diffs that Cafe uses to store gradients with respect to that data um and that gives you two and then you actually have four because there's a CPU and a GPU version of each of those things so you have data you have diffs you have CPU you have GPU so there's actually four n-dimensional tensors per blob the next important uh class that you need to know about in Cafe is the layer um and a layer is a sort of a function s of similar to the ones you wrote on the homeworks that receives some input blobs Cafe calls inputs bottoms and then produces output blobs um that Cafe calls top blobs so the idea is that your layer will receive pointers to the bottom blobs with the data already filled in and then it'll also receive a pointer to the top blobs and it'll and in forward pass it'll be expected to fill in the values for the data elements of your um top blobs um then in the backward pass the layers will compute gradients so they will expect to receive um a pointer to the top blobs um with the gradients and the activations filled in and then they'll they'll also receive a pointer to the bottom blobs and fill in the gradients for the bottoms um and layer is this a pretty well structured uh abstract class that you can go and I've had I have the the links for the source and the header file here and there's a lot of sub classes that Implement different types of layers um and uh a like I said common Cafe problem there's no really good list of all the layer types um you pretty much just need to look at the code and see what types of CPP files there are um the next thing you need to know about is a net so a net just combines a bunch of layers um a net is basically a directed a cichet graph of layers and is responsible for running the forward and the backward methods of the layers in the correct order um so this is uh you probably don't need to touch this class ever yourself but it's kind of nice to look at to get a flavor of how everything fits together um and the final class that you need to know about is a solver so the solver um is you know we had this thing called solver on the homework that was really inspired by Cafe so a solver is intended to um dip into the net run the net forward and backward on data um actually update the parameters of the network and handle checkpointing and resuming from checkpoints and all that sort of stuff and in Cafe solver is this abstract class and different update rules are implemented by different uh sub classes so there's for example a stochastic gradient descent solver there's an atom solver there's an RMS prop solver all of that sort of stuff and again just to see what kinds of options are available you should at the source code um so this kind of gives you a nice little overview of how these things all fit together that this whole thing on the right would be a net the net contains um in the green boxes blobs each blob contains data and diffs the red boxes are layers that are connecting blobs together and then the whole thing would get optimized with a solver so Cafe makes heavy use of this funny thing called protocol buffers um any of you guys ever interned a Google yeah Fair number so you guys know about this um but protocol buffers are this um almost like a binary strongly typed Json I sort of like to think about it that are used very widely inside Google for serializing data to disk and for um passing it over the network so protocol buffers um there's this prototo file that defines the different kinds of fields that different types of objects have so in this example there's a person that has a name and an ID and an email and this lives in apro file um Proto files uh give Define a type of a class and you can actually serialize instances to human readable. prototxt files um so for example this fills in the name it gives you the ID it gives you the email and this is an instance of a person that can be saved into this text file um then protuff includes this compiler that actually lets you generate um classes in various programming languages to access these data types um so you can after running the protuff compiler on this Proto file it produces classes that you can then import in Java and C C++ and python done and go and just about everything um so actually Cafe makes wide use of these Proto of these protocol buffers and they use them to store pretty much everything in Cafe so um Cafe like I said to understand you need to read the code to understand Cafe and Cafe has this one giant file called cafe. prototo that just defines all of the protocol buffer types that are used in Cafe so this is a gigantic file it's it's I think it's a couple thousand lines long but it's actually pretty well documented and is I think the most upto-date documentation of what all the layer types are what the options for those layers are how you specify every all the options for solvers and layers and Nets and all that so I'd really encourage you to check out this file and read through it if you um have any questions about how things work in Cafe and just to give you a flavor um on the left here this shows you uh this defines the net parameter which is the type of protocol buffer that Cafe uses to represent Nets and on the right is this solver parameter which um use which Cafe uses to represent solvers so the net parameter um or so the solver parameter for example takes uh a reference to a net and it also includes things like learning rate and how often to checkpoint and other things like that um right so when you're working in Cafe actually it's pretty cool you don't need to write any code in order to train models um so when working with Cafe you generally have this four-step process so first you'll convert your data um and especially if you just have an image classification problem you don't have to write any code for this you just use one of the existing binaries in that Cafe ships with um then you'll Define your your net file that you'll do by just writing or editing one of these Proto txts um then you'll Define your solver which again will just live be a protot txt text file that you can just work with in a text editor and then you'll pass all of these things to um this existing binary to train the model and that'll spit out your trained Cafe model to disk that you can then use for other things so even if you wanted to train resnet on imag net you could just follow this simple procedure and train a giant Network without writing any code so that's really cool um and so step one generally you'll need to convert your data so Cafe uses I know we've talked a little bit about hdf5 as a format for storing pixels on disk continuously and then reading from them efficiently um but by default Cafe uses this other file format called lmdb um so there's a if you if all you have is a bunch of images each image with a label then you can call then Cafe just has a script to convert that whole data set into a giant lmdb that you can use for training um so just to give you an idea the way it's this is really easy you just create a text file that has the path to your images and separated by the label and you just pass it to your Cafe script um wait a couple hours if your data set's big and you end up with a giant lmdb file on disk and uh if you're working with with something else like hdf5 um then you'll have to create it yourself probably um so Cafe does actually have a couple other options for reading data in there's uh this data layer the window data layer for for detection um it actually can read from hdf5 um and there's an option for reading stuff directly from memory that's especially useful with the python interface but at least in my point of view all of these types of other methods of reading in data to Cafe are a little bit second class citizens in the cafe ecosystem and lmdb is really the easiest thing to work with so um if you can you should probably try to convert your data into lmdb format with Cafe so step two for Cafe is to Define your net object um so like I said you'll just write a big Proto txt to Define your net so here this is a this just defines a simple model for logistic regression you can see that I did not follow my own advice and I'm reading data out of an hdf5 file here um then I have a fully connected layer which uh is an called inner product in Cafe um then right so that fully connected layer tells you the number of classes and how to initialize the values and then I have um uh a softmax loss function that reads the labels and uh produces loss and gradient from the outputs of the fully connected layer so a couple things to point out about this file are that one every layer um you typically includes some blobs which uh to store the data and the gradients and the weights and the layers blobs and the layer itself typically have the same name so that can be a little bit confused using um another thing is that um a lot of these layers will have two blobs one for weight and one for bias and actually in this n in right in here you'll Define the learning rates for those two blobs um so that's learning rate and regularization for both the weight and the bias of that layer another thing to note is that uh to specify the number of output classes is just the numb output on this uh fully connected layer parameter and finally um the the quick and dirty way to freeze layers in Cafe is just to set the learning rate to zero for that for the blobs Associated to that layer's weights or biases another thing to point out is that um for resnet and other large models like googlet this can get really out of hand really quickly so Cafe doesn't really let you define like compositionality so for resnet they just repeat the same pattern over and over and over in the prototxt file so the resnet protot txt is is almost 7,000 lines long um so you could write that by hand but in in practice people tend to write little Python scripts to generate these things automatically um so that's that's a little bit gross um if you if you want to fine-tune a network rather than starting from scratch then you'll typically uh download some existing protot txt and some existing weights file and work from there so the way you should think about it is that the prototxt file that we've seen here before defines the architecture of the network and then the the pre-trained weights live in this uh Cafe model file that's a binary thing and you can't really inspect it but the way that it works is it's basically key value pairs um where it matches n where the in the inside the cafe model it matches these names um that are scoped to layers so this uh fc7 we would be the would be the the weight corresponding to this final fully connected layer in alexnet so then uh when you want to F tune on your own data um when you start up Cafe and you load a cafe model and a protot TX it just tries to match the key value pairs of names and weights between the cafe model and the protot txt um so if the names are the same then your new network gets initialized from the values in the protot txt which is really really useful and convenient for fine tuning um but if the layers if the names don't match then it those layers actually get initialized from scratch so this is how for example you can reinitialize the output layer in Cafe so uh to be a little bit more concrete uh if you've maybe downloaded an image model then this layer is going to this final fully connected layer that's outputting class scores will have a thousand outputs um but now maybe for some problem you care about you only want to have 10 outputs so you're going to need to reinitialize that final layer and reinitialize it randomly and fine-tune the network so the way that you do that is you need to change the name of the layer um in the protot txt file um to make sure that it's actually initialized randomly and not reading from the from the from the cafe model and if you forget to do this then it'll actually crash and it'll give you a weird error message about the shapes not aligning CU it'll be trying to store this thousand dimensional weight Matrix into this 10-dimensional thing from your new file and it won't work um so the next step when working with Cafe is to define the solver um the solver is also just a Proto txt file you can see all the options for it in that giant. prototo file that I gave a link to um but it it'll look something like this for alexnet maybe so that will Define your learning rate and your learning rate Decay and your regularization how often to checkpoint everything like that um but these end up being less much less complex than uh than these Proto txts for the networks um this Alex net guy is just maybe 14 lines um although what you will see sometimes in practice is that if people want to have um sort of complex training pipelines where they first want to train with one learning rate in certain parts of the network then they want to train with another learning rate certain parts of the network that you might end up with a Cascade of different solver files and actually run them independently where you're sort of fine-tuning your own model in separate stages using different solvers so once you've done all that then you just train your model so um if you if you followed my advice and just used lmdb and written all these things then you just call this binary um that is that exists in Cafe already so here you just uh pass it your solver and your protot txt um and your pre-trained weights file if you're fine-tuning and it'll run maybe for a day maybe for a long time and just checkpoint and save things to disk and you'll be happy one thing to point out here is that you specify which GPU it runs on um this is zero indexed but you can actually run in CPU only mode by setting this flag to to negative one um and actually recent uh sometime in the last year Cafe added data parallelism to let you split up mini batches across multiple gpus in your system um so you can actually add uh multiple gpus on this flag and if you just say all then Cafe will automatically split up many batches across all the gpus on your machine so that's really cool you've done multi-gpu training without writing a single line of code pretty cool um Cafe oh yeah question yeah I think I think that'll be kind of so the question is how would you go about doing some more complex initialization strategy where you maybe want to initialize the weights um from a pre-train model and use those same weights in multiple parts of your network and the and the answer is that you probably can't do that with the simple mechanism um you can kind of munge around the weights in Python and that's probably how you'd go about doing it um right so I think we've mentioned this before that Cafe has this really great model Zoo you can download um lots of different types of pre-trained models on imag net and other data sets so this this model zoo is really top-notch you've got alexnet and BGG you've got resnets up there already um pretty much uh lots and lots of really good models are up there so that's that's a really really strong part point about Cafe that it's really easy to download someone else's model and run it on your data or fine-tune it on your data um Cafe has a python interface like I mentioned I I since there's so many things to cover I don't think I can dive into detail here but um as is kind of part for the course in Cafe there's not really really great documentation about the python interface so you need to read the code um and the whole the the python interface for Cafe is mostly defined in these two in these two files um this CPP file uses boost python if you've ever use that before to to wrap up some of the C++ classes and expose them to Python and then in this uh py file um it actually attaches additional methods and gives you a more pythonic interface so um if you want to know what kinds of methods and data types are available in the cafe python interface your best bet is just just uh read read through these two files and they're not too too long so it's it's pretty easy to do um yeah so the python interface in general is is pretty useful it lets you do um maybe crazy weight initialization strategies if you need to do something more complex than just copy from a pre-train model um it also makes it really easy to just get a network and then run it forward and backward on uh with nump from numpy arrays so for example um you can Implement things like deep dream and class visualizations similar to that you did on the homework you can also do that quite easily using the the python interface uh on Cafe where you just need to take data and then run it forward and backward through different parts of the network um the python interface is also quite nice if if you just want to extract features um like you have some data set you have some pre-train model and you want to extract features from some part of the network and then maybe save them to disk uh maybe to an HTF file for some Downstream processing so that's quite easy to do with the python interface um you can also actually Cafe has a kind of a new feature where you can actually Define layers entirely in Python um but this is I've never done it myself but it's it seems cool it seems nice but the downside is that then those layers will be CPU only so we talked about communication bottlenecks between the CPU and the GPU that um if you write layers in Python then every forward and backward pass you'll be incurring uh uh overhead on that transfer um although one nice place where python layers could be useful is custom loss functions so that's maybe something that uh you could keep in mind so the quick overview of Cafe C pros and cons is that really from my point of view if all you want to do is kind of train a simple basic feedforward Network especially for classification then Cafe is really really easy to get things up and running you don't have to write any code yourself you just use all these uh pre-built tools and it's quite easy to run um it has a python interface which is uh quite nice for using a little bit more for a little bit more complex use cases but it can be cumbersome when things get really crazy when you have these really big networks like resnet or googlet especially with repeated module patterns it can be tedious um and for things like RN like recurrent networks where you want to share weights between different parts of the network um can be kind of com kind kind of cumbersome in Cafe um it is possible but it's probably not the best thing to use for that um and the other downside the other big downside from my point of view is that when you want to Define your own type of layer in Cafe you end up having to write C++ and Cuda code so that's not doesn't give you a very quick development cycle so it's kind of a lot of kind of painful to write new layers in Cafe so that's that's our Whirlwind Whirlwind tour of Cafe so if there's any quick questions um yeah how would you do something like cross validation cross validation in Cafe um I so in the train Val Proto txt you can actually Define uh a a training phase and a testing phase so generally you'll write a like a train valve Proto txt and a deploy protot txt and the deploy will be used at on the test set and the test phase of the train Val Proto txt will be used for validation okay that's that's all there is to know about Cafe so uh the next one is torch so torch is really my personal favorite so I have a little bit of bias here just to get that out in the open that I've pretty much been using torch almost exclusively for my own projects in the last year or so um so torch is originally from NYU um it's written in C and in Lua and it's used a lot at Facebook and Deep Mind especially um I think also a lot of F at Twitter use torch um so one of the big things that freaks people out about torch is that you have to write in Lua um which I had never Lear I had never heard of or used before starting to work with torch but it actually isn't too bad um that Lua is this highly uh this high level scripting language that is um really intended for embedded devices so it can run very efficiently um and it's a lot very similar to JavaScript in a lot of ways um so another cool thing about Lua is that because it's meant to be run on embedded devices that you can actually do for Loops are really fast and torch um you know how in Python if you write a for loop it's going to be really slow um that's actually totally fine to do in torch because it actually um uses just in time compilation to make these things really fast um and torch is or Lua is most similar to JavaScript in that it is a functional language functions are first class citizens and it's very common to P pass callbacks around to different parts of your code um Lua also uh is has this idea of prototypical inheritance um where there's sort of one data structure which in Lua is a table which you can think of as being very similar to an object in JavaScript um and you can Implement things like object-oriented programming using prototypical inheritance in a similar way as you would in JavaScript um and one of the down one of the downsides is actually the standard library in Lua is um kind of annoying sometimes and things like handling strings and whatnot can be kind of cumbersome um and maybe most annoying is that it's one indexed so all of your intuition about for Loops will be a little bit off for a while um but other than that it's pretty easy to pick up and I gave a link here to this uh website claiming that you can learn Lua in 15 minutes um it might be a little bit of an over uh they might be overselling it a little bit but I think it is pretty easy to pick up Lua and start writing code in it pretty fast so the main idea behind torch is this tensor class so um you guys have been working in numpy a lot on your assignments and the way the assignments are kind of structured is that um the numpy array gives you this really easy way to manipulate data in whatever way you want and then you can use that numpy array to build up other abstractions like neural net libraries and whatnot but really the numpy array um just lets you uh manipulate data numerically in whatever way you want in complete flexibility um so if you recall then maybe here's a here's an example of some numpy code that should be very familiar by now where we're just Computing a simple forward pass of a two- layer Rayo Network so um oh maybe black wasn't the best choice here but um we're uh we we doing we're Computing some some constants we're Computing some weights we're getting some random data and we're doing a matrix multiply a ra Lo and another Matrix multiply um so that's that's very easy to write in numpy and actually this has almost a onetoone translation into torch tensors so now on the right this is the exact same code but using torch tensor in Lua so here we're defining our batch size input size and all that um we're defining our weight which are just torch tensors um we're getting a random input Vector we're doing a forward pass um this is doing a matrix multiply of torch tensors um this cmax is a element wise maximum so that's a Ru and then we can compute scores using another Matrix multiply so in general pretty much any kind of code you're used to writing in numpy is pretty easy pretty much has almost a one by one line by line translation into using torch tensors instead um so you'll also remember in numpy that it's really easy to swap and use different data types um we talked about this ad nauseum in the last lecture but at least in numpy um to switch to maybe a 302 bit floating Point all you need to do is cast your data to this uh other data type and it turns out that that's very very easy to do in torch as well that our data type is now this this string and then we can easily cast our data to um another data type but here's where here's the so this next slide is the real reason why torch is infinitely better than numpy um and that's that the GPU is just another data type um so when you want to write when you want to run code on the GPU in torch um you use this you import another package and you have another another data type which is torch. Cuda tensor and now you cast your tensors to this other data type and now they live on the GPU and running any kind of numerical operations on Cuda tensors just runs on the GPU so it's really really easy in torch to just write generic tensor scientific Computing code to run on GPU and be really fast um so this like I said these tensors are really you should think of them as similar to numpy arrays um and there's a lot of documentation on the different kinds of methods that you can work with in tensors um up here on GitHub um this documentation isn't super complete but it's it's not bad so you should take a look at it so um the next but in practice you end up not really using the tensors too much in torch instead use this other package called NN for neural networks so NN is this pretty thin wrapper that actually defines a neural network package um just in terms of these tensor a in terms of these tensor objects um so you should think of this as being like a beefier more industrial strength version of the homework code base where you have this um this T this ND array uh this tensor abstraction and then you implement a neural net library on top of it um in a nice clean interface so here's this same two-layer Rao Network us using the NN package so we Define our Network as a sequential so it's going to be a stack of of sequential operations um it's going to we're going to first have a linear which is a fully connected layer from our input Dimension to our hidden Dimension we're going to have a Rao and another linear um now we can actually get the the weights and gradients um in in one tensor for each using this get parameters method so now weights will be a single torch tensor that will have all the weights of the network and grad weights will be a single torch tensor for all of the all of the gradients um we can generate some random data now to do a forward pass we just call Net the forward method on the on the net object using our data this gives us our scores um to compute our loss we have a separate uh Criterion object that is our loss function so we compute our loss by calling the forward method of the Criterion um now we've done our forward pass very easy and for our backward pass we first set our gradients to zero um call a backward on the loss function and then a back on the network um now this has updated all of the gradients for the network in the grad prams so we can just make a a gradient step very easily so this would be um multiplying the grad weights by the opposite of the learning rate and then adding it to the weights so that's a simple uh gradient descent update uh right so that's that's all of the right so that would have been maybe a a little bit more clear but we have a n we have weights grad weights we have our loss function uh we get random data run forward run backward make an update um and as as you might expect from looking at the tensor it's quite easy to make this thing run on GPU so to run one of these networks on the GPU um we import a couple new packages C torch and qn which are Cuda versions of everything um and then we just need to cast our Network and our loss function to this other data type um and we also need to cast our data and labels and now this whole network will run and train on the GPU so it's it's pretty easy now in uh what what is that like 40 lines of code we've written a fully connected Network and we can train it on the GPU um but one problem here is that we're just using vanilla gradient descent um which is not so great and as you saw in the assignments other things like atom and RMS prop tend to work much better in practice so to solve that torch gives us the optim package so optim is quite easy to use um again we just import a new package up here um and now what changes is that uh we actually need to Define this call back function so before we were just calling forward and backward excl uh explicitly ourself instead we're going to Define this call back function that will run the network forward and backward on data and then return the loss and the gradient and now to make an update step on our Network we'll actually pass this callback function to this uh atom method from the optim package so this this is maybe a little bit awkward um but we you now we can use uh any kind of update rule using just a couple lines of change from what we had before um and again this is very easy to add to run on a GPU by just casting everything to Cuda right so as we saw in Cafe um Cafe sort of implements everything in terms of Nets and layers and Cafe has this really hard distinction between a net and a layer um in torch they don't we don't really draw this distinction everything is just a module um so the entire network is a module and also each individual layer is a module um so so uh modules are just classes that are defined in Lua that that are implemented using the the tensor API so these modules are since they're written in Lua they're quite easy to understand so linear is the fully connected is the the fully connected layer in Lua um and this is the Constructor you can see it's just setting up uh tensors for the weight and the bias um and because this tensor API in torch lets us easily run the same code on GPU and CPU then all of these layers will just um be written in terms of the tensor API and then easily run on both devices so these modules need to implement a forward and backward so for forward uh they decided to call it update output um so here's the example of the the update output for the fully connected layer um there's actually a couple cases they need to deal with a couple different cases here to deal with mini batch versus non- minib batch inputs but other than that this should be quite easy to read um for for the backward pass there's a pair of methods update grad input which um receives the upstream gradients and computes the gradients with respect to the input and again this is just implemented in the tensor API so it's uh very easy to to understand it's just a it's just the same type of thing you saw in the homework um and we also Implement an accumulate grad parameters which computes the gradients with respect to the weights of the network um as you saw in the Constructor the weights and the biases are held in instance variables of this module and accumulate grad parameters will you receive gradients from upstream and accumulate gradients of the parameters with respect to the Upstream gradients um and again this is very simple just using the tensor API um so torch actually has a ton of different modules available um the documentation here can be a little bit out of date but if you just go on GitHub you can see all the Lua files that give you all the goodies to play with um and these actually get updated a lot so just to point out a couple these uh these three were just added in last week so torch is always adding new modules to that you can add to your networks which is pretty fun um but when these existing modules aren't good enough it's actually very easy to write your own so because uh you can just Implement these things using these tensor using the tensor API and just implement the forward and backward it's not much harder than implementing layers on the homeworks so here's just a small example um this is a stupid module that just takes its input and multiplies it by two um and you can see we implement the update output uh and the update grad input and now we've implemented a new layer in torch in just 20 lines of code and then that's really e and then it's very easy to use in other code um just import it and now you can add it to networks and and so on and the really cool thing about this is because this is just the tensor API you can do whatever kind of arbitrary thing you want inside of these forward and backward um if you need to do for Loops or um complicated imperative code or any or maybe stochastic things for Dropout or batch normalization then any kind of any whatever kind of code you want on the forward and backward path you just implement it yourself inside these modules so it's usually very very easy to implement um your own new types of layers in torch um so torch uh but of course using individual layers on their own Isn't So useful we need to be able to stitch them together into larger networks so for this torch uses containers um we already saw one in the previous example which was this sequential container so a sequential container is just um a stack of modules that all where one receives the output from the previous one and just goes in a linear stack so that's probably the most commonly used um another one you might see is this uh par is this um concat table so maybe if you have an input and you want to apply different two different modules to the same input then the concat table lets you do that and you receive the output as a list um another one you might see is a parallel table if you have a list of inputs and you want to apply different modules to each element of the list then you can use a a parallel table table for that sort of a construction um but when things get really complicated um so actually those those uh containers that I just told you should in theory be easy to um be possible to implement just about any topology you want but it can be really hairy in practice to wire up really complicated things using those containers so um torch provides another package called NN graph that lets you hook up um container hook up things in more complicated topologies pretty easily um so here's an example if we have maybe if we have three inputs and we want to produce one output and we want to produce them with this pretty simple update rule um that corresponds to this type of computational graph that we've seen uh many times in lecture for different types of problems um so you could actually implement this just fine using um parallel and sequential and concat table but it could be kind of a mess um so when you want to do things like this it's very common to use NN graph instead so this NN graph code is is quite easy so here this function is going to build a module using NN graph and then return it so here we import the NN graph package um and then inside here this is a bit of funny syntax so this is actually not a tensor this is defining a symbolic variable so this is saying that our our tensor object is going to receive X Y and Z as inputs and now here we're actually doing symbolic operations on those inputs so here we're saying that um a we we want to have a point-wise addition of X and Y um store that in a we want to have point-wise multiplication of a and z and store that in B and now a pointwise addition of A and B and store that in C and again these are not actual tensor objects these are now sort of symbolic references that are being used to build up this computational graph in the back end um and now we can actually return a module here where we say that our module will have inputs X Y and Z and outputs c um and this n n.g module will actually give us uh an object conforming to the module API that implements this computation so then after we build the module we can construct um concrete torch tensors and then feed them into the module that will actually compute the function um so torch actually is uh quite good at pre-train models there's a package called load Cafe that lets you um load up many different types of pre-train models um from Cafe and it'll convert them into their torch equivalents um so you can load up uh the cafe protot txt and the cafe model file and it'll turn into a giant stack of sequential layers um load Cafe is not super General though and only works for certain types of networks um but in particular load Cafe will let you load up alexnet and Cafe Net and V so they're probably the some of the most commonly used um there are also a couple different implementations that let you load up googlet into into torch to let you load up pre-trained Google net models into torch um and actually very recently Facebook went ahead and reimplemented the residual networks um straight up in torch and they released pre-train models for that so between alexnet Cafe Net vgg googlet and resnet I think that's probably everything you need um all the pre-train models that most people want to use um another point is that because torch is using Lua we can't use pip to install packages and there's another a very similar idea called L rocks lets you easily install new packages on update packages um that's quite very easy to use um and this is kind of just a list of some packages that I find very useful in torch so there are CN bindings you can read and write to HTF files um you can read and write Json um there's this funny one from Twitter called autograd that is a little bit like theano which we'll talk about in a bit but I haven't used it but it's kind of cool to to look at um and actually Facebook has a pretty useful uh library for torch as well that implements um fft convolutions and also implements data parallel and model parallelism um uh so that's pretty a pretty nice thing to have so a very typical workflow in torch is that um you'll have some pre-processing script often in Python that'll pre-process your data and dump it onto some nice format in disk um usually hdf5 for big things and Json for little things then you will I'll typically write a train. Lua that'll read from the hdf5 and train the model um and optimize the model and save checkpoints to disk um and then usually I'll have some evaluate script that loads up the train model and then does it for something useful so a case study for this type of workflow is this project I put up on GitHub like a week ago um that implements uh character level language models in torch so here there's a pre-processing script that converts text files into hdf5 files there's a training script that um loads from hdf5 and trains these recurrent networks and then there's a sampling script that loads up the checkpoints and let you generate text so that's that's kind of like uh my typical workflow in torch so the quick pros and cons I would say about torch are that it's Lua this is a big turnoff for people but I don't think it's actually that big a deal um it's definitely less plug andplay than Cafe so you'll end up writing a lot of your own code typically um which maybe is a little bit more overhead but also gives you a lot more flexibility um it has a lot of modular pieces that are easy to plug in play um and the the standard library because it's all written in Lua is quite easy to read and quite easy to understand and there's a lot of pre-train models um which is quite nice um but unfortunately it's it's a little bit awkward to use for recurrent networks in general um so when you want to have one when you want to have multiple modules that share weights with each other you can actually do this in torch but it's it's kind of brittle and you can run into subtle bugs there um so that's that's probably the biggest caveat is that recurrent networks can be tricky um any any quick questions about torch yeah when you mention that uh it's easy to uh it's efficient to run uh for Loop so are you saying for example recolution net layer we used to have four layers in the na version and in the fast one you just findall uh so do you still gain any benefit to yeah yeah you definitely still gain benefit from writing more efficient code but it's not oh so the question was about um how how bad are for Loops in Lua um and python is interpreted right so that's that's really why for Loops are really bad in Python because it's interpreted and every for Loop is actually doing quite a lot of memory allocation another behind the scenes um but if you've ever used JavaScript then Loops in JavaScript tend to be pretty fast because the runtime actually jits the um compiles the code on the fly down to native code um so Loops in JavaScript are really fast and LU and Lua actually has a similar mechanism where it'll sort of automatically um and magically compile the code for you into native code so your Loops can be really fast but definitely um writing custom vectorized code still can give you a lot of speed up all right so we've got um now maybe half an hour left to cover two more Frameworks so we're running out of time so next up is theano uh so theano is from yosua benio's group at University of Montreal and it's really all about computational graphs so we saw a little bit in NN graph from torch that computational graphs are this pretty nice way to stitch together big complicated architectures um and theana really takes this idea of computational graphs and runs with it to the extreme um and it also has some highle libraries Caris and lasagna that we'll touch on as well so here's the same computational graph we saw in the context of NN graph before and we can actually walk through an implementation of this in in theano so you can see that in here we're importing theano and the theano tensor object um and now here we're defining X Y and Z as symbolic as symbolic variables this is actually very similar to the NN graph example we saw just a few slides ago so that these are actually not numpy arrays these are sort of symbolic objects in in the in the computation graph um then we can actually compute these outputs um symbolically so x y and z are these symbolic things and we can compute a b and c just using these overloaded uh operators and that'll be building up this computational graph in the back end then once we've built up our computational graph we actually will want to be able to run certain parts of it on real data so we call this theo. function thing so this is saying that we want to take our function will take inputs X Y and Z and it'll produce outputs c um and this will return an actual python function that we can evaluate on real data and I'd like to point out that this is really where all the magic in Theo is happening that when you call it a function it can be doing crazy crazy things it can simplify your computational graph to make it more efficient it can actually um symbolically div derive gradients and other things and it can actually generate native code so um when when you call function it can it actually sometimes compiles code on the fly to run officially on the GPU so all the magic in theano is really coming from this from this little innocent looking statement in Python but there's a lot going on under the hood here um and now once we've gotten this magic function through all this crazy stuff then we can just run it on actual numpy arrays so here we're instantiating XX y y and ZZ as act as actual numpy arrays and then we can um just evaluate our function and pass in these actual numpy arrays to get the values out and this is doing the same thing as doing these computations explicitly in Python except that the that the theano version could be much more efficient due to all the magic under the hood and the theano version actually could be running on the GPU if you have that configured um but unfortunately we don't really care about Computing things like this we want to do neural Nets so here's an example of a simple two- layer R net in theano so the idea is the same that we're going to declare our inputs um but now instead of just X Y and Z we have our inputs in X um our labels in Y which are a vector um and our two weight matrices W1 and W2 so we're just sort of setting up these symbolic variables that will be um elements in our computational graph now for our forward pass we it looks kind of like numpy but it's not um these are operations on these symbolic objects that are building up the graph in the back end so here we're Computing activations with um this dot method that that is Matrix multiply between these symbolic objects um we're doing a railu using this this Library function and we're doing another Matrix multiply and then we can actually compute the loss uh the probabilities and the loss using a couple other Library functions and again these are all operations on these symbolic objects that are building up the the computational graph so then we can just compile this function um so our function is going to take our data our labels and our two weight vector and our two weight matrices as inputs and as outputs that'll turn the loss in a scaler and our classification scores in a vector and now we can run this thing on real data just like we saw on the previous slide where we can instantiate some actual numpy arrays and then pass them into the function um so this is great but this is only the forward pass we actually need to be able to train this network and compute gradients so here um we just need to add a couple lines of code to do that so this is the same as before we're we're defining our symbolic variables for our input and our weights and so forth and we're combining we're running the the same forward pass as before to compute the loss um to compute the loss symbolically now the difference is that we actually can do symbolic differentiation here so this dw1 and dw2 we're telling theano that we want those to be the gradient of the the gradient of the loss with respect to those other symbolic variables W1 and W2 so this is really cool um theano just lets you take arbitrary gradients of any part of the graph with respect to to any other part of the graph and now introduce introduce those as new symbolic variables in the graph so that you can really go crazy with that but here in