CS231n Winter 2016: Lecture 15: Invited Talk by Jeff Dean

Key Takeaways

I'd like to point out that what I'll be presenting today is partly my work in collaboration with others and sometimes I'm presenting work done by people in my group uh that I wasn't really involved in but it's joint work with many many people you'll see lots of names throughout the talk so uh take that with a grain of salt uh so what I'm going to tell you about is kind of how Google got to where it is today in terms of using deep learning in a lot of different places um the project that uh I'm involved in actually started in 2011 when Andrew in was spending one day a week at Google and I happen to bump into him in the micro kitchen and uh and I said oh what do you were doing he's like I don't know but I haven't figured out yet but big neurl Nets are are interesting I'm like oh that's cool and turns out I'd done an underg thesis on Parallel training of neural Nets like ages ago um I won't even tell you how long ago uh back kind of in the first exciting period of neural Nets uh and I always kind of really liked the the computational model they provided uh but that time it was a little too early like we didn't have big enough data sets we didn't have enough computation to really make them sing and uh Andrew kind of said oh well it'd be interesting to train big Nets so I'm like okay that sounds fun uh so we kind of collaboratively started uh the brain project to push uh the size and scale of neural Nets that we could train uh and in particular we were really interested in using big data sets and large amounts of computation to tackle perception problems and language problems um um Andrew then went off and founded corsera and kind of drifted away from Google but uh since then we've been doing a lot of interesting work in both kind of research areas in a lot of different domains you know one of the nice things about net is they're incredibly applicable to many many different kinds of problems as I'm sure you've seen in this class uh and we've also deployed production systems using neural Nets in a pretty wide variety of different products so I'll kind of give you a sampling of some of the research some of the production aspect some of the systems that we have built underneath the covers uh including kind of um some of the implementation stuff that we do in tensorflow to make these kinds of models run fast uh and I'll focus on neurl Nets but a lot of the techniques are more applicable than just neurl Nets uh tensor flow you can train lots of different kinds of reinforcement algorithms or other kinds of um you know other kinds of machine learning models than just n Nets okay can everyone hear me excellent someone in the back give a thumbs up so uh one of the things I really like about uh the team we've put together is that we have a really broad mix of different kinds of expertise so we have people who are really experts at machine learning research you know people like Jeffrey Hinton uh other people like that uh we have large scale distributed systems Builders I kind of consider myself in that more in that mold and then we have people kind of with a mix of those skills and often some of the projects we work on you collectively put together people with these different kinds of expertise and um um collectively you do something that none of you could do individually because often you need both kind of large scale systems thinking and sort of clever machine learning ideas so uh that's always fun and you often kind of pick up and learn new things from other people which is great uh so rough outline actually this is from an old deck uh so you know you can kind of see the progress of how Google has been applying deep learning across lots of different areas this is sort of when we started the project and we started collaborating with a speech team a bit and started doing it with some kind of early computer vision kinds of problems uh and as we had success in some of those domains other teams at Google would say hey uh I have a problem too and like they would come to us or we would go to them and say hey we think this could help with your particular problem and over time we've kind of gradually uh not so gradually expanded the set of teams and areas that we've been applying these kinds of problems to um and you see the breadth of different kinds of it's not like it's only computer vision problems uh so that's that's kind of nice we're continuing to grow which is good uh and part of the reason for that broad spectrum of things is that you can really think of neural Nets as uh these nice really Universal systems that you can put lots of different kinds of inputs into you can put Lots get lots of different kinds of outputs out of them uh with you know slight differences in the model you try but in general the same fundamental techniques work pretty well across all these different domains uh and it gives state-of-the-art results as I'm sure you've heard about in this class and lots of different areas you know pretty much any computer vision problem uh any speech problem these days uh starting to be more the case in lots of language understanding areas uh lots of kind of other areas of science like uh drug Discovery are starting to have interesting neet models that are better than Alternatives uh yeah I like them they're good along the way we've kind of built two different generations of our underlying system software for training and deploying neural Nets uh the first uh was called disbelief we published a paper about that in nips 2012 uh it had the advantage that it was really scalable like the first one of the first uses we put to it was doing some unsupervised training I'll tell you about in a minute which used 16,000 cores to train a big net with a lot of parameters uh was good for production use but it wasn't super flexible for research like it was kind of hard to express kind of weirder more esoteric kinds of models reinforcement learning algorithms would be hard to express in it had this kind of much more layer driven approach with up and down messages and it worked well for what it did uh but we kind of took a step back about a year and a little bit ago and started building our second generation system tensorflow which is based on what we learned with the first generation and what we learned from looking at other sort of uh available open source packages um and we think it's retained a lot of the good features of disbelief but also made it pretty flexible for a wide variety of research uses uh We've open sourced it which uh I gather you heard about uh so one of the really nice properties of n Nets I grabbed this from a particular paper because it had graphs on both scaling the size of training data and how accuracy increases and also scaling the size of the neurl net and how accuracy increases um the exact details aren't important you can find these kinds of Trends in hundreds of papers but one of the really nice properties is if you have more data and you can make your model bigger generally scaling both of those things is even better than scaling just one of them uh you need a really big model in order to capture kind of the more subtle trends that appear in larger and larger data sets you know any neural net will capture kind of obvious Trends or obvious kinds of patterns but the more subtle ones are ones where you need a bigger model to capture kind of that extra suance of subtlety um and that requires a lot more computation so we focus a lot on scaling uh the computation we need and being able to train big models on big data sets uh right okay so one of the first things we did in this project was we said oh unsupervised learning is going to be really important and we had a big focus on that initially um uh quackle and others uh uh said what would happen if we did unsupervised learning on random Youtube frames so the idea is we're going to take 10 million random Youtube frames single frames from a bunch of random videos and we're going to essentially train an auto encoder everyone knows what an auto encoder is yeah so this is like a fancy multi-level autoencoder uh you know in this one we're just trying to reconstruct the image and that one we're trying to reconstruct the representation here from the representation there and so on um and we used uh 16,000 cores we didn't have gpus in the data center at the time so we compensated with like throwing more CPUs at it um and we used async SGD which I'll talk about in a minute for optimization uh it actually had a lot of parameters because it was not convolutional this was prior to uh convolutions being all the rage so we said well we'll have local receptive Fields but they won't be convolutional and we'll learn like separate representations for this part of the image and this part of the image um which is kind of an interesting twist I think it' be actually an interesting experiment to redo this work but with convolutional uh parameter sharing that would be kind of cool uh in any case the representation you learned at the top after like nine layers of of these non-con local receptive Fields had 60,000 neurons at the top level and one of the things we thought might happen is it would learn kind of highle feature detectors so in particular you're putting in pixels but it could it learn high level Concepts uh so we had a data set that was half faces and half not face faes and we found looked around for neurons that were uh good selectors of whether or not the image the test image contain a face and we found several such neurons the best one uh that are those are some of the sample images that cause that neuron to get the most excited um and then if you look around for what stimulus will cause that neuron to get the most excited uh there's creepy face guy um and and so that's kind of interesting like we had no labels in the image in the data set at all that when were training and a neuron in this model has picked up on the fact that faces are things uh I'm going to get excited when I see kind of a Caucasian face from head-on um it's YouTube so we also have a cat neuron we had a data set that was half cats and half not cats and this is uh average Tabby I call him uh and then you can take that unsupervised model and and start a supervised training task so in particular at this time we were uh training on the image net 20,000 class task which is uh not the one that most image net results are reported on that one's a thousand classes so this is trying to distinguish an image from one of 20,000 classes it's a a much harder task um and when we trained and then looked around at what kinds of uh images cause different top level neurons to get excited you see that they're picking up on very high level Concepts um you know yellow flowers only or water foul pizzas I like pizza um and this pre-training actually increased the stud of- the art accuracy on that particular task a fair amount at that time uh then we kind of lost our excitement about unsupervised learning because supervised learning works so darn well um and so we started working with a speech team who uh at the time was uh had a non net based acoustic model so you essentially trying to go from a small segment of audio data like 150 millisecs and you're trying to predict what sound is being uttered in the middle 10 milliseconds um and so uh we just decided to try a eight layer fully connected nlet and then predict one of 14,000 triphones at the top uh and that worked phenomenally well um basically you could train it pretty quickly and it gave a huge reduction in Word like this is one of the people on speech team said that's like the biggest single Improvement they've seen in their 20 years in speech research uh and that launched uh uh as part of the Android uh voice search system uh in 2012 I think um so one of the things we often do is find that we have a lot of data for some tasks but not very many very much data for other tasks and so for that we often deploy systems that uh make use of multitask and transfer learning in various ways so let's look at an example where we use this in speech so obviously with English we have a lot of data and we get a really nice low word error rate lower is better um for Portuguese on the other hand at that time we didn't have that much training data we had 100 hours of Portuguese data and so the word error rate is a lot worse uh which is bad um so one of the first and most simple things you can do which is kind of what you do when you take a model that's been pre-trained on imet and applied to some other problem where you don't have as much data is you just start training with those weights rather than totally random weights uh and that actually improves your word error rate for Portuguese a bit because there's enough similarities in the kinds of features you want for speech and general regardless of language um a more complicated thing you can do is actually jointly train models that share a bunch of parameters across all languages or in this case all all um European languages I think is what we used um and so there you see uh we're jointly training on this data and we actually got a pretty significant Improvement even over the just copying the data to the Portuguese model um but surprisingly we actually got a small Improvement in English because in in total across all the other languages we actually almost doubled the amount of training data we were able to use in this model um compared to just English alone so basically languages without much data all improved a lot languages with a lot of data improved even a little bit um and then we had a language specific top layer we did a bit a little little bit of fiddling to figure out you know does it make sense to have two language specific top layers or one um sadly these are the kinds of human guided choices you you make in notet uh the production speech models have evolved a lot from those fairly simple feed forward models um they use now lstms to deal with time Dimension they use convol convolutions to make them invariant to different frequencies so there was a paper published here you you know you don't necessarily need to understand all the details but there's a lot of more complexity in the kinds of model and it's it's using much uh more sophisticated recurrent models and convolutional models um a recent trend has been that you can use lstms completely end to end so rather than having an acoustic model and then a language model that kind of takes the output of the acoustic model and then is trained somewhat SE separ you can go directly from audio waveforms to producing transcripts a character at a time um and I think that's going to be a really big Trend uh both in speech and more generally in a lot of computing systems you often have today a lot of systems are kind of composed of a bunch of subsystems each perhaps with some machine learned pieces and some kind of hand coded pieces and then usually a big pile of gooey code to glue it all together and often all those separately developed pieces have impediments to optimization right like you optimize your subsystem in the context of some Metric but that metric might not be the right thing for the final task you care about which might be transcribing speech correctly um so having a much bigger single system like a single neural net that goes directly from audio waveform all the way to the end objective you care about transcription uh and that you can then optimize end to end through and there's not sort of a lot of handwritten code in the middle of that uh is going to be a big Trend I think you'll see that here you'll see that in machine translation lots of other kinds of domains uh so this class is all about convolutions so we have tons of vision problems that we've been using various kinds of convolutional models for um you know the big uh excitement around convolutional Nets well first it started with Y Yan lon's check reading uh convolution net and then it kind of like um subsided for a while and then Alex keski oio seter and Jeffrey hinton's paper in 2012 which like blew the other competitors out of the water in the image net 2012 challenge uh using an Earl net um I think put those things on everyone's map again and saying wow we really should be using these things for vision because they work really well and the next year there were something like 20 20 of the entries or something used neur Nets whereas previous viously it was just Alex um uh we've had a bunch of people at Google looking at various kinds of architectures for doing better and better image net classifications so the Inception architecture has like this complicated module of like different size convolutions that are all kind of concatenated together and then um you kind of replicate those modules a bunch of times and very you end up with a very deep nurl net uh that turns out to be quite good at uh recognition um there's been some slight additions to that and slight changes to make it even more accurate you know have you seen a slide like that and like actually I think we had a slide like this with Inception before already okay yeah so I I was lazy and only took my slides from a older thing I have you told the story about Andre sitting down and hand labeling images I think I think a little bit okay so anyway Andre decided since he was helping to administer the image net contest he would sit down and subject himself to 100 hours of train with like a training test training test split and like is that an Australian Shepherd dog I don't know and he also I think convinced one of his labmates to do it but they weren't as diligent so he did about 120 hours of training on images and his his poor labmate got tired after 12 hours or something so he got 5.1% error and the labmate got I think 12% error yeah so funny Sor about that we originally tried to do it on emble of our entire lab and and had a group meeting where we all just tried to like sit down and evaluate human test error on imet but without training um we did really really poorly so that's why Andre like got fed up with how bad we all did and went home for the weekend and came back and was really good at image now yeah yeah 112 hours later or whatever anyway he has a great blog post about it I encourage you to check it out he has a lot of parameters though so typical humans are like uh you know 80 Australian connections I'm going to give him him a 100 um uh one one point about these models is that models with a small number parameters fit well on like mobile devices so Andre doesn't fit well on a mobile phone but the general Trend other than Andre is like smaller numbers of parameters compared to alexnet mostly alexnet had like these two giant fully connected layers at the top not that giant but a lot of parameters and later work just kind of did away with those for the most part and so they've used um you know a smaller number of parameters but more floating Point operations as they reuse convolutional parameters more uh which is good for um fitting them on phones uh we released as part of the tensorflow update uh a pre-trained Inception model that you can use uh there's a nice tutorial about it uh there's Grace Hopper although we think it's military uniform which is not terribly inaccurate um one of the nice things about these models is they're really good at doing very fine grain classifications I think one of the things that is in Andre's blog is that the computer models are actually much much better than people at distinguishing exact breeds of dogs but humans are better at uh often picking out a small you know if if the label is pingpong ball and it's like a giant scene of people playing pingpong humans are better at that because the models tend to focus on things with more pixels um uh if if you train models with the right kinds of data you know it'll generalize while these scenes look nothing alike but they actually you know will both get labeled as meal if you're training data is represented uh well uh they make kind of sensible errors which is kind of nice uh you know it's not a snake but you understand why it might have said that and I know it's not a dog but I actually had to think carefully if the front animal there is a is a donkey or a goat I'm not still not entirely sure any votes yeah right it's hard um so one of the production uh uses we've put these kinds of models too is Google photo search so we launched Google photo uh product and you can search the photos that you've uploaded without tagging them at all you just type ocean and all of a sudden all of your ocean photos show up uh so for example this user posted publicly hey I and they posted a screenshot hey I didn't tag these and I typed statue and they just showed up or see drawing you know Yoda this is a tough Yoda because it's got a lot of texture compared to most Yodas so we're pretty pleased to retrieve macrom made Yodas um uh we have a lot of kind of other kinds of more uh specific visual tasks like essentially one of the things we want to do in our street view imageries we have these cars that drive around the world and take pictures of all the the roads and the street scenes and then we want to be able to read the text that we find so first you have to find the text um and well one of the first things you want to do is find all the addresses and then help improve the maps and once you do that you want to like read all the other text uh so you can see that it does we have a model that does a pretty good job of predicting at a pixel level which which pixels contain text or not um and it does pretty well in that well first of all it finds lots of text and the training data had different kinds of characters sets represented so it has no problem recognizing Chinese characters English characters or Roman Latin characters uh it does pretty well with like different colors of of text and different fonts and sizes and some of them are very close to the camera some are very far away and it do just fine and this is data from just human labeled drawn polygons around pieces of text and then they transcribed it and then we have an OCR model that we also trained uh uh we've been kind of gradually releasing other kinds of products so we just launched a cloud Vision set of apis you can do lots of things like label images this is meant for people who don't necessarily want to want or have machine learning expertise they just kind of want to do cool stuff with images you want to be able to you know say oh that's a Running Scene uh you want to be able do the OCR and find uh text in any image that you upload so you just basically give it an image and say I'd like to run OCR and lab generation on this image and off it goes uh so people have been pretty happy with that uh you know internally people have been thinking of more creative uses of how to use computer vision so essentially now that computer vision sort of really actually works compared to five years ago um this is something that our our um our Geo team that processes satellite imagery put together and released which is basically a way of predicting the slope of roofs from multiple satellite views of that because we like have you know every few months we get new satellite imagery and so we have multiple views of the same location and we can predict what the slope of the roof is given all those different views of the same location and how much sun exposure it'll get and then predict you know if you were to install solar panels on your house how much uh energy could you generate or that kind of thing kind of cool you know it's like a small random thing that you can do now that Vision Works uh okay um so this class has been I assume mostly about Vision so I'm going to talk now about other kinds of problems like language understanding um one of the most important problems is search obviously so we care a lot about search um and in particular if I do the query car parts for sale I'd like to determine which of these two documents is more relevant and if you just look at the surface forms of the word that first document looks pretty darn relevant because it has like lots of the words occur and lots of red um but actually the second document is much more relevant uh given that um and we'd like to be able to understand that so how much have you talked about embeddings um they actually implemented an lstm cap sh model on awesome so you know about embeddings great embeddings are fantastic uh so I will go quickly but basically you want to represent words or things in high dimensional things that are sparse map them into a dense space some 100 dimensional or thousand dimensional space so that you can now have um things that are near each other uh and have similar meanings will end up near each other in these high dimensional spaces so for example you'd like porpus and dolphin to be very near each other in this High dimensional space because they're quite similar words and have similar meanings they share the same sense of ppose no okay maybe not and SeaWorld should be kind of nearby and camera and Paris should be pretty far away um and you can train embeddings in lots of different ways one is to have it kind of as the first thing you do when you're feeding data into an lstm an even simpler thing is uh a technique my former colleague tamas mov came up with that we published a paper about where essentially it's called the word Toc model and essentially you pick a window of words maybe 20 words wide you pick the center word and then you pick another random word you try to use the embedding representation of that Center word to predict the other word um and then you can train that whole thing with back propop so essentially you adjust the weights in the softmax classifier and then in turn you through uh uh back propagation you you make little adjustments to the embedding representation of that Center word so that next time you'll be able to better predict the word parts from automobile uh and it actually works right like that's one of the really nice things about embedding is given enough training data you get really phenomenal representations of words so these are the nearest neighbors for these three different uh words we actually had short phrases as vocabulary items in this particular run so tiger shark you can think of as one one embedding vector and these are the nearest neighbors so it it got the sense of sharkness uh car is interesting right like you see why this is useful for search because you have things things that people often hand code in information retrieval systems like plurals and stemming and like some kind of simple synonyms but here you just see like oh auto car automobile pickup truck racing car passenger car dealership is kind of related you just see that it has this this right concept of a nice kind of smooth representation of car rather than explicitly only the letters c match that um and it turns out that if you train using the word DEC approach the directions turn out to be meaningful in these high dimensional spaces so not only is proximity interesting but uh directions are interesting so it turns out if you look at uh capital and country pairs you go roughly the same direction and distance to get from a country to its corresponding capital or vice versa um for any country capital pair um and you also kind of see some semblance of other structures so this is the settings map down to two Dimensions via principal components analysis so uh um and you see kind of interesting structures around verb tenses regardless of the verb um which means you can solve analogies like Queen is to King as woman is to Man by doing some simple Vector arithmetic so you're literally just looking up the embedding vector and then adding the difference to get to that point approximately that point um so we've been uh in collaboration with the search team we launched kind of one of the biggest search ranking changes in the last few years um we called it rank brain uh it's essentially just a deep neurl net that uses embeddings and a bunch of of neural net layers to give you a score for how relevant this document is for this particular query um and it's the third most important search ranking signal now uh out of hundreds of such signals um that's called uh smart reply was a a little collaboration we did with the Gmail team where essentially uh replying to mail on your phone kind of sucks because typing is hard and so we wanted to have a system where often You can predict what would be a good reply uh just looking at the message so we have a small Network that predicts is that a likely to be something that I can have a short tur response to if yes then we activate much bigger lstm based model and this is a message one of my colleagues received who was working on the project said from his brother he said we wanted to invite you to join us for an early Thanksgiving blah blah blah blah blah please bring your favorite dish RSVP by next week so then the model uh predicts count us in will be there or sorry we won't be able to make it um which is great if you get a lot of email it's fantastic um although your replies will be somewhat tur although you can edit them which is nice ni um you know we can do interesting things like uh this is a a mobile app that actually runs in airplane mode so it's actually running the models on the phone um and it's actually got a lot of interesting things some of which I didn't entirely realize so um you're essentially using the the camera image you're detecting text in it you're finding what the words are doing OCR on it you're then running it through a trans model you configure it in a particular mode this is just cycling through different languages but normally you'd set it on Spanish mode and it would only show you Spanish um but uh the thing I didn't realize is there's actually an interesting font selection problem in how to like choose what font to show for the output so anyway it's kind of cool good if you're traveling to interesting placees I'm actually going to Korea uh on Saturday so I am I'm looking forward to using my translate app because I don't speak Korean um so one of the things we do a bit of work on is reducing inference cost uh there's like nothing worse than this feeling that wow my model is so awesome it's great it's it's just sad that it drains my phone's battery in 2 minutes or you know I can't afford the computation to run it at you know High QPS in my data center even though I have gobs and machines um so there's lots of tricks you can use uh in particular the simplest one is inference is generally much more forgiving of even much lower Precision uh um computation than training so for inference we usually find we can quantize all the weights to 8 Bits or even less 8 Bits is just a nice quantity that CPUs like to deal with really you could do six bits probably but that doesn't help you that much um so that gives you like a nice 4X memory reduction in storing the parameters and also gives you 4X computation efficiency because you can use CPU Vector instructions to do four multiplies instead of one 32bit floating Point multiply uh so that's really good I'm going to tell you about kind of a cuter more exotic way of getting more efficiency out of a mobile phone uh this is a technique called distillation that uh Jeffrey Hinton Oriol vignols and I worked on um so suppose you have a really really giant model the problem I just described where you have this Fantastic Model you're really pleased with uh maybe you have an ensemble of those and now you want a smaller cheaper model with almost the same accuracy so here it is you're a giant expensive model uh you feed this image in and it gives you fantastic predictions like 0. N5 Jaguar I'm pretty sure um and I'm definitely sure that's not a car 10 the minus 6th for car 04 you know I'm hedging a bit it could be a lion right uh so that's what a really accurate model will do um so the main idea unfortunately we later discovered that rich Carana in 2006 had published a similar idea in a paper called Model compression uh so The Ensemble or your giant accurate model implements this interesting function from input to Output so if you forget the fact that there's some structure there and you just try to use the information that's contained in that function um how can we transfer the knowledge in that really accurate function into a smaller representation of the function um so when you're training a model typically what you do is you feed in an image like this and then you give it targets to try to achieve and you give it the target one jaguar and zero for everything else um I'm going to call that a hard target so that's kind of the the ideal your model is striving to achieve and you give it you know hundreds of thousands or millions of training images and it drives to approxim made all these you know one hot vectors for all the different images um and in actual fact it doesn't quite do that because it gives you this nice Pro probability distribution over different images over different uh classes for the same image um so let's take our giant expensive model and one of the things we can do is we can actually soften that distribution a bit and this is what Jeffrey Hinton calls uh dark knowledge sounds fancier um but if you soften this by essentially dividing all the logistic units by a temperature T that might be like five or 10 or something you then get a softer representation of this probability distribution where you say oh okay it's a Jaguar but also kind of head your bets a little and call it a bit of a lion maybe even less of a cow still call it definitely not a car um and that's something you can then uh use and this full distribution conveys a lot more information about the image about the function that's being implemented by this large Ensemble right because the Ensemble is trying to hedge its bets and do a really good job on giving you a probabil probability distribution over that image so then you can train the small model so normally when you train you just train on the hard targets but instead you can train on some combination of the hard targets plus the soft targets um and the training objective is going to try to mat match some function of those two things um so this works surprisingly well so here's an uh an experiment we did on a large speech model so we started with a model that classified 58.9% of frames correctly uh that's our big accurate model and now we're going to use that model to provide soft targets for a smaller model that also gets to see the hard targets and we're going to train it on only 3% of the data so the new model with the the soft targets gets almost that accuracy 57% um and with just hard targets it drastically overfits and it gets to 44.5% accurate and then goes south um so soft targets are really really good regularizer and the other thing is that because these stof targets have so much information in them compared to just a single one and a bunch of zeros you train much much faster you get to that accuracy in like phenomenally shter amounts of time uh so that's that's pretty nice and you can do this approach with like giant ensembles mapping into one size model of that Ensemble you can do from a large model into a smaller one um somewhat underappreciated technique okay uh let's see uh so one of the things we did when we thought about building tensorflow was we kind of took a step back from where we were and we said what do you really want in a research system so you want a lot of different things and it's kind of hard to balance all of those things uh but really one of the things you really care about if you're a researcher is ease of expression I want to be able to take any old research idea and uh try it out yeah a quick question about the last slide so you said that you were able to train a lot faster and get it up to 57 how much smaller model um I don't remember the exact details there in this archive paper uh it was considerably smaller I think like instead of thousand wide fully connected layers it was like 600 or 500 wide which is actually big difference but check in that paper for the details I'm probably misremembering uh right and then you want to be able to take your research idea scale it and run experiments quickly you want to be able to run it portably on both you know data centers and on phones uh it's nice to be able to reproduce things and you want to go from a good research idea to a production system without having to rewrite in some other system uh so that's how we kind of the main things we were considering when building tensorflow uh we open sourced it as I'm as you're aware um I kind of already said that our first system wasn't as flexible uh so the core bits of tensor flow are we have a notion of different devices uh it is portable so it runs on a bunch of different operating systems we have this core graph execution engine and then on top of that we have different front ends where you express the kinds of computations you're trying to do we have a C++ frontend which most people don't use and then we have the python frontend which most of you are are probably more familiar with um where most thing but there's nothing preventing people from putting other languages on there we want it to be fairly language neutral um so there's uh some work going on to put a go from front end on there other kinds of languages we think that'll be good uh and you want to be able to take that model and run it on a pretty wide variety of different platforms uh the basic computational model is a graph I don't know how much you talked about this in your overview of little bit okay so there's a graph things that flow along the edges are tensors so arbitrary n dimensional arrays with a primitive type like float or int um unlike pure data flow models there's actually state in this graph so you have things like biases which is a variable and then you have operations that can update things that have persistent state so you can go through the whole graph compute some gradient and then adjust the biases based on the gradient um the graph goes through a series of stages uh one important stage is deciding given a whole bunch of computational devices and the graph where are we going to run each of the different nodes in the graph in terms of computation uh so for example here we might have a CPU in blue and a GPU card in green um and we might want to run the graph in such a way that all the expensive computation happens on the GPU um so actually as an aside this placement decisions are kind of tricky we allow users to provide hints that guide this a bit um and then given the hints which are not necessarily hard constraints on an exact device but might be something like uh you should really try to run this on a GPU or um uh place it on task s and I don't care what device um and then we want to basically minimize the time through the graph subject to all kinds of other constraints like the memory we have available on each GPU card or on CPUs uh uh I think it'd be interesting to actually use an nlet with some reinforcement learning because you can actually measure an objective here of you know if I place this node and this node and node in this way how fast is my graph and I think that would be a pretty interesting reinforcement learning uh problem uh once we've made decisions of where to place things then we insert these send and receive nodes which essentially encapsulate all the communication in the system so basically when you want to move a tensor from one place to another there's a send node that'll kind of just hold on to the tensor until the receive node checks in and says hey i' really love that data for that tensor uh and you do this for all the edges that cross device boundary uh and you have different implementations of send and receive pairs depending on the device so you have for example if the gpus are on the same machine you can often do RDMA directly from one gpu's memory to the other uh if they're on different machines then you do a cross machine RBC your network might support RDMA across the network uh in that case you would just use you know directly reach into this other gpus memory on this other machine and grab it uh you can Define new operations and kernels pretty easily uh the session interface is essentially how you run the graph um and typically you run you set up a graph once and then you run it a lot so that allows us to kind of have the system do a lot of optimization and decisions about um essentially how it wants to place computation nodes and perhaps do some experiments on like does it make more sense to put it here or here uh because it's going to amortize that over lots and lots of run calls uh the single process configuration everything runs in one process and it's just uh sort of simple procedure calls in a distributed setting there's a client process a master process and then a bunch of workers that have devices and the master you client says I'd like to run this subgraph the master says oh okay that means I need to talk to process one and two and tell them to do stuff um you can feed and fetch data and that means that I might sort of have a more complex graph but I only need to run run little bits of it because I only need to run the parts of the computation that the outputs you asked for are uh are needed based on what you asked for uh we focus a lot on being able to scale this to distributed environment uh we actually one of the biggest things when we first open sourced tensorflow we hadn't quite carved aart a open sourceable distributed implementation um so that was GitHub issues number 23 which got filed within like a day of our release uh said Hey where's the distributive version uh we did the initial release last Thursday so um that's good it'll get better packaging but at the moment you can kind of hand configure multiple processes with the names of the other processes uh involved IP addresses and ports um we're going to package that up better in the next couple of weeks uh but that's there which is good uh and the whole reason to have that is that you want much better turnaround time for experiments so if you're in the mode where you're training and experiment iteration is kind of minutes or hours that's really really good if you're in the mode of like multiple weeks that's kind of hopeless right like more than a month you you generally won't even do it or if you do you're like oh my experiment is done why did I do that again um so we really emphasize a lot in our group just being able to make it so people can do uh experiments as fast as as reasonable uh so the two main things we do are model parallelism and data parallelism I'll talk about both you've talked about this a little bit or a little bit but not in too much detail okay great so the best way you can decrease training time is decrease the step time uh so one of the really nice properties most neural Nets have is there's lots and lots of inherent parallelism right like if you think about a evolutional model there's lots of parallelism uh at each of the layers because all the spatial positions are mostly independent you can just run run them uh in parallel on different devices uh the problem is figuring out how to communicate how to distribute that computation in such a way that uh communication doesn't kill you um a few things help you so one is local connectivity like convolutional neural Nets have this nice property that they're generally looking at like a 5x5 patch of data below them and they don't need anything else uh and the neuron next to it has a lot of overlap with the data it needs for for that first neuron um you can have towers with little or no connectivity between the towers so every few layers you might communicate a little bit but mostly you don't uh the original Alex net paper did that so it essentially had two separate towers that mostly ran independently on gpus two different gpus and occasionally exchanged uh some information you can have specialized part to the model that are active only for some examples uh there's lots of ways to exploit parallelism so when you're just naively compiling Matrix multiply code with GCC or something it'll probably already take advantage of instruction parallelism uh present on Intel CPUs cross cores you can use thread parallelism and and spread Things That Way Cross devices uh communicating between gpus is often pretty limited so you have like a factor of 30 to 40 better bandwidth to the local GPU memory than you do to like another GPU cards memory on the same machine and across machine boundaries is generally even worse so it's pretty important to kind of keep as much data local as you can and avoid needing too much uh communication bandwidth but model parallelism the basic idea is you're just going to partition the computation of the model somehow uh maybe spatially like this uh maybe layer by layer um and then in this case for example the only communication I need to do is at this boundary you know some of the data from partition two is needed for the input of that partition one but mostly all the data is local uh the other technique you can use for speeding up convergence is data parallelism so in that case you're going to use many different replicas of the same model structure and they're all going to collaborate to update parameters so uh in some shared set of uh servers that hold the parameter State uh speed UPS depend a lot on the kind of model you know it could be 10 to 40x speed up for 50 replicas uh sparse models with like really large embeddings for every vocabulary word no demand generally you can support more parallelism because most updates only update a handful of the embedding uh entries you know if you have a sentence that has like 10 unique words in it out of a million and you can have millions and millions or thousands of replicas uh doing lots of work so the basic idea in data parallelism is you have these different model replicas you're going to have this centralized system that keeps track of the parameters that may not just be a single machine it may be a lot of machines because you need a lot of network bandwidth uh sometimes to uh keep all these model replicas fed with parameters so that might you know in our big setups that might be 127 machines at the top uh and then you know you might have 500 replicas of the models down there uh and before every model replica does a mini batch it's going to grab the parameters so it says okay U7 machines give me the parameters and then it does a computation of a random mini batch and figures out what the gradient should be it doesn't apply the gradient locally it sends the gradient back to the parameter servers parameter servers then update the current parameter values uh and then before the next step we do the same thing uh fairly Network intensive depending on your model things that help here are models that don't have very many parameters convolutions are really nice in that respect lstms are nice in that respect because you're essentially then reusing every parameter lots and lots of times so you already reuse it you know however big your batch size is on the model so if your batch size is 128 you're going to bring a parameter over you're going to use it 128 times for all the different elements in the batch but if you have a convolutional model now you're going to get an additional factor of reuse of maybe like 10,000 different positions in a layer that you're going to reuse it and in an lstm if you unroll 100 time steps you're going to reuse it 100 times uh just for the unrolling um so those kinds of things that have models that have lots of computation and fewer parameters to sort of drive that computation generally will work better in data parallel environments um now there's an obvious issue depending on how you do this so one way you can do this is completely asynchronously every model replica is just sitting in a loop and fetching the parameters doing a mini batch Computing gradient sending it up there and if you do that asynchronously then the gradient it computes may be completely stale with respect to the where the parameters are now right you've computed it with respect to this parameter value but meanwhile 10 other replicas have caused the parameters to meander over to here and now you apply the gradient that you thought was for here to this value um so it makes theoreticians incredibly uncomfortable they're already uncomfortable because it's completely non-convex problems um but the good news is it works uh up to a certain level it would be really good to understand the conditions under which you know this works in a more theoretical basis but in practice it does seem to work pretty well the other thing you can do is do this completely synchronously so you can have one driving Loop that says okay everyone go they all get the parameters they all compute gradient and then you wait for all the gradients to show up and do something with the gradients average them or add them together um and that effectively just looks like a giant batch so if you have R replicas that looks like you know R times each individual one's batch size which uh sometimes works you kind of get diminishing returns from larger and larger batch sizes but the more training examples you have the more tolerant you are of a bigger batch size generally so if you have a trillion training examples you know batch size of 8,000 isort of okay if you have a million training examples batch size of 8,000 is not so great uh right I think I said this there's even more complicated choices where you can have like M asynchronous groups of end synchronous replicas uh right I've said that um convolutions and recurrent models are good because they reuse the parameters a lot uh so data ism is actually really really important for almost all of our models uh that's how we get to the point of training models in like half a day or a day generally um so uh you know you see some of the rough kinds of setup for use um and here's an example training graph of image net model one GPU 10 gpus 50 gpus and there's the kind of speed UPS you get right like sometimes these graphs are deceiving like the difference between 10 and 50 is doesn't seem that big like because the lines are kind of close to each other sort of but in actual fact the difference between 10 and 50 is like a factor of 4.1 or something so that that doesn't look like a factor of 4.1 difference does it but it is um anyway yeah the way you do it is you you like see where that one crosses 6 and see where that one crosses 6 uh okay um so let me show you some of the slight tweaks you make to tensorflow models to exploit these different kinds of parallelism one of the things we wanted was for these kinds of parallelism uh Notions to be pretty easy to express so one of the things we like about tensorflow is it Maps pretty well to the kind of the things you might see in a research paper so it's not you know you don't have to read all that but it's not too uh different than what you would see in the research paper which is kind of nice so that's like a simple lstm cell uh this is the sequence to sequence model uh thater Oro vs and qule published in nips 2014 uh we're essentially trying to take an input sequence and map it to an output sequence uh this is a really big area of research it turns out these kinds of models are applicable for lots and lots of kinds of problems uh there's lots of different groups doing interesting and active work in this area um um yeah so here's just some examples of recent work in the last year and a half in this area from lots of different labs around the world uh you've already talked about captioning I guess cool yes so instead of a sequence you can put in pixels who knew uh so you put in pixels you run it through a CNN that's your initial State and then uh you can generate captions it's pretty amazing so if you'd asked me five years ago can a computer do that I would have said I don't think so not for a while uh here we are uh it can actually do and the nice thing is it's a gen generative model so you can generate different sentences by exploring the distribution you know I think both of those are nice captions it's not quite as sophisticated as the human one you'll often see this uh uh one of the things is if you if you train the model a little bit it's really important to train your models to convergence because like that's not so good but if you train that model longer it's the same model it just got a lot better same thing here right train that is sitting on the tracks yes that's true but that one's that one's better but you still see the human has a lot more sophistication right like they know that they're cross the tracks near a Depot and that's sort of a more subtle thing that the models don't pick up on uh another kind of cute use of lstms you can actually use them to solve all kinds of cool graph problems so uh uh oral vignols um meor Fortunato and nde jetley did this work which uh you start with that of points and then you try to predict um the traveling salesman tour for that that works best or the convex Hull or uh Delon triangulation of graphs um it's kind of cool you know it's just a sequence to sequence problem where you feed in the sequence of points and then the output is the right set of points for whatever problem you care about uh I already talked about smart reply okay so lstms so once you have that lstm cell code that I showed you on there uh you can unroll it in time 20 time steps uh let's say you wanted four layers per time step instead of one well you would make a little bit of change to your code and you would do that um now you have four layers of computation instead of one uh one of the things you might want to do is run each of those layers on a different GPU so that's the change you would make to your tensorflow code to do that um and that then allows you to have a model like this so this is my sequence these are the different deep lstm layers I have per time step um and after the first little bit I can start getting more and more gpus kind of involved in the process uh and you essentially pipeline the entire thing uh there's a giant soft Max at the top which you can split across gpus pretty easily um so that's model parallelism right we've now got six gpus in this picture we actually use eight we split the self Max cross four gpus um and then so every replica would be eight GPU cards on the same machine all kind of humming along and then you might use data parallelism in addition to that to train you know a bunch of eight GPU card replicas uh to train quickly uh we have this notion of qes so you can kind of have tensorflow graphs that do a bunch of stuff and then stuff it in a que and then later you have another bit of tensorflow graph that starts with deqing some stuff and then uh does some things so one one example is you might want to prefetch inputs and then like do the jpeg decoding to convert them into sort of arrays and maybe do some whitening and cropping and random crop selection and then stuff them in a cube and then uh you can then DQ on say different GPU cards or something uh we also can group similar examples so for translation work uh we actually bucket by length of sentence so that your batch has a bunch of examples that are all roughly the same sentence length all 13 to 16 word sentences or something uh that just means we then need to only execute exactly that many unrolled steps rather than you know arbitrary Max sentence length uh it's good for randomization and shuffling so we have a shuffling cue you can just stuff a whole bunch of examples and then uh get random ones out uh oh yeah I don't know what I'm saying uh data parallelism right so again we want to be able to have many replicas of this thing uh and so you make modest amounts of changes to your code we're not quite as happy with this amount of change but this is kind of what you would do there's a supervisor that has a bunch of things you now say there's parameter devices uh and then prepare the session and then each one of these runs a local Loop um and you now keep track of how many steps have been applied globally across all the different replicas and as soon as the cumulative sum of all those is big enough you they all exit so asynchronous training looks kind of like that you have three separate client threads driving three separate replicas all with parameter devices so one of the big simplifications from disbelief to tensor flow is we don't have a separate parameter server notion anymore we have uh tensors and variables variables that contain tensors and they're just just other parts of the graph uh and typically you map them onto a small set of devices they're going to hold you parameters but it's all kind of unified in the same framework whether I'm sending a tensor that's parameters or activations or whatever it doesn't matter uh this is kind of a synchronous view I have one client and I just split my batch across three replicas and add the gradient and apply them uh neural Nets turn out to be pretty tolerant of reduced Precision so uh you know you convert to fp6 um there's actually an i e standard for 16bit floating points now floating Point values now uh most CPUs don't quite support that yet so we uh implemented our own 16-bit format which is essentially we have a 32-bit float and we Lop off two bytes of mantisa um uh and you should kind of do stochastic probabilistic rounding but we don't so sort of okay it's just noise uh and then you can convert it to 32 bits on the other side by filling in zeros uh it's it's very CPU uh friendly uh oh typo well um so model and data parallelism in conun Combined really let you train models quickly and that's what this is all really about is being able to take a research idea try it out on a large data set that's representative of a problem you care about figure out did that work figure out what the next set of experiments is um it's pretty easy to express in tensorflow the data parallelism we're not so happy with for asynchronous parallelism uh but in general it's it's not too bad um we've open source tensor flow uh because we think that'll make it easier to share research ideas uh we think you know having um lots of people using the system outside of Google is a is a good thing they'll improve it and bring ideas that we don't necessarily have uh it makes it pretty easy to deploy machine Learning Systems into real products because you can go from a research idea into something running on a phone relatively easily uh the community of tensorflow users outside Google is growing which is nice uh they're doing all kinds of cool things so I picked a few random examples of things people have done that are posted on GitHub uh this is one that's like a Andre has this this convet JS which runs uh neural Nets in your browser using JavaScript and one of the things he has is a little game that uses reinforcement learning where the the yellow dot learns to I think it learns to eat the oh yeah learns to eat the green dots and avoid the red dots so someone reimplemented that in tensorflow and actually added orange dots that are really bad um and someone implemented this really nice paper from University of tbri and the max pun Institute I don't know if have you seen this work where you take an image a picture and typically a painting and then it renders the the picture in the style of that painter um and you end up with cool stuff like that um so someone implemented that you know there's a a character RNN model the Kiros is a popular sort of higher level library that make it easier to express neural Nets um someone implemented the the neural captioning model in flow um there's a effort underway to translate it into Mandarin uh it's cool it's great uh the last thing I will talk about is the brain Residency program so we've started this program it's a bit of an experiment this year so uh this is more as an FYI for next year because our applications are closed we're actually selecting our final candidates this week and then the idea is the people will spend a year in our group doing deep learning research and the hope is um they'll come out and have published a couple of papers on archive or submitted to conferences and learn a lot about doing uh sort of interesting machine learning research um and you know we're looking for people for next year obviously uh that are strong in you know anyone taking this class probably fits fits the bill uh we'll reopen applications in the fall so if you're graduating like next year this could be a good opportunity uh there you go there's a bunch more reading there uh start here because I did a lot of work in the tensorflow white paper to make the whole set of references clickable and so you can click your way through to 50 other papers okay that's all I have and I'm done early 160 slid [Applause] yeah so it seems like models like smart reply get better uh with the more data you have and Google probably has the biggest data set of email in the world but people in front of might be like uncomfortable with having their private email being used for this sort of thing so how do you guys handle this sort of privacy accuracy yes how do you that so those kinds of things are actually tricky and we have an actually a pretty extensive detailed process for things that are you know talking about you know using a user's private data for these kinds of things so uh for smart reply essentially all the replies that were that it ever will generate are things that have been said by thousands of users um so the input to the model for training is an email which is typically not said by thousands of people but the only things we'll ever suggest are things that are generated in response by you know sufficient number of unique users to uh protect the privacy of the users so that's the kind of things you're thinking about when designing products like that and there's actually a lot of care and thought going into you know we think this would be a great feature but how can we do this in a way that that ensures that people's privacy is protected yeah um I want to ask a question with respect to some of the work we did to knowledge distillation I remember from the paper that um guys we're working on this massive gft um data set and you guys at some point decided okay we need a lot of different specialist networks in order to deal with from some of the more difficult aspects of the classification and at some point in the paper you guys mentioned that you know you considering the possibility of then distilling all these different Specialists into single larger Network I think there's um you know in iclr there's already some work done by um R Group on the use of multitask learning in such a format where they have multiple specialist for different games and justar a single thing but that was still on a relatively small scale and I was wondering you know have you guys considered has there been some further work done on really large scale distillation of specialist n so we haven't pursued the distillation work as much as um we probably should have should uh it's just kind of been one of the things on the back burner compared to all the other things we've been working on I do think um the notion of Specialists so I didn't talk about that at all but essentially we had a model that was a sort of a arbitrary image net classification model or like jft which is like 177,000 glasses or something it's an internal data set so we trained a good General model that could deal with all those classes and then we found interesting confuse confusable classes that AO algorithmically like all the kinds of mushrooms in the world and we would train Specialists on data sets that were enriched with only mushroom data primarily and then occasional random images um and we could Train 50 such models that were each good at different kinds of things and get pretty significant accuracy increases um uh at the time we we were able to distill it into a single model pretty well but we haven't really pursued that too much turns out just the mechanics of then training 50 separate models and then distilling them is a bit unwieldy yeah I had a question about the the part you of the big model teaching the little model but isn't a bit worrying to think that a small model when we optimize it with classic techniques we only get like 40% accuracy and when we optimize it with other techniques we get 60% accuracy because because like it seems the capacity of the model is enough to solve the tax but we don't know how to optimize properly so isn't it like doesn't it mean that we fail at optimizing everything in some way um I I will first say I think the area of optimization is a ripe one for exploration and further research because as you say this clearly demonstrates that we're I mean it's a different objective we're telling model to do right we're telling it to use this hard label or use this hard label and also get this incredibly Rich gradient which says like here's a 100 other signals of information so in some sense it's an unfair comparison right it you're telling it a lot more stuff about every example in that case so in some sense it's not so much an optimization failing it's maybe we should be fig figuring out how to feed richer signals than just a single binary label to our model I think that's probably an interesting area to pursue we've thought about ideas of having a big Ensemble of models all training collectively and sort of exchanging information in the form of their predictions rather than in their parameters because that might be a much cheaper more Network friendly way of of collaboratively training on a really big data set where you each train on 1% of the data or something and swap predictions could could you just do a simple thing where you take the the captions like this is a holding a banana can you convert that to a more Rich non one hot classification label so something like banana 50% % just convert the entire data then you train on Rich your labels yeah I mean I think all these kinds of ideas are worth pursuing um the the captioning work is interesting but it tends to We tend to have many fewer labels with cap then we have images with sort of labels like cheetah or Jaguar um at least that are prepared in a clean way I think actually on the web there's a lot of images with sentences written about them the trick is identifying which sentence is about which image got yeah yeah um to what extent do your models like train onl ches yeah so it depends on the problem some problems you know you don't need to really train online like speech recognition is a good example it's not like human vocal cords change that often uh the words you say change a little bit so query distributions tend to be not very stationary right like the words everyone collectively says tomorrow are pretty similar to the ones they say today but subtly different uh like Long Island Chocolate Festival might suddenly become more and more prominent over the next two weeks or something uh and those kinds of things you know you need to be cognizant of the fact that you want to capture those kinds of effects and one of the ways to do it is to train your model in an online manner sometimes it doesn't need to be so online that you like get an example and immediately update your model but you know depending on the problem every 5 minutes or 10 minutes or hour or day is sufficient for most problems um but it is pretty important uh to do that for non-stationary problems like ads or or search queries or things that change over time like that yeah you mentioned that rank brain was the third most important signal for search what are one and two if you can tell us I can't say yes yeah your training data uh yeah I mean noise and training data sets actually happens all the time right like even if you look at the image net examples occasionally you'll come across one and you're like actually I was just sitting in a a meeting with some people who are working on visualization techniques and one of the things they were visualizing was uh cfar input data and they had this kind of cool representation of all the cfar examples all mapped down to like 4x4 pixels each on on their screen so 60,000 images and like you could kind of pick things out and select and sort and you're like oh here's one that like the model predicted with high confidence but it got wrong and it said airplane as uh the model said airplane and you look at the image and it's an airplane and the label is not airplane you're like oh I understand why I got it wrong um so it it's you know you want to make sure your data set is as clean as possible because training on noisy data is generally not as good is clean data but on the other hand expending too much effort to clean the data is often more more effort than it's worth so you kind of do some filtering kinds of things to you know throw out the obvious bad stuff and generally more noisy data is often better than less clean data Bas just put theel hope ites depends on the problem but but certainly that's one thing to try and then if you're unhappy with the results then investigate why other questions okay cool all right thank you [Applause]