How to Use Tensorboard (LIVE)

Key Takeaways

your life hello world it's suraj we started a little late because I was having some tensor board trouble which I still am having but it's all good because when I say I'm going to go live I'm gonna go live so today we are going to talk about tensor board so tensor board is the tool that comes with tensor flow so it's built in so if you if you installed tensor flow tensor board comes with it and the goal of tensor board is to help you visualize your data there's it can help you optimize your data it can help you debug it it can help you look at ways that you can improve it and there's also the plugins feature which is coming soon so I can't wait for that but yeah it's a pretty cool tool and I admit I haven't used it much but I and that's because I think the use case for it really shows itself when your data starts to get really complex that is your models so when your models start to get really big that's when tensor board becomes very useful and one of the reasons I'm demoing it now so I already made a video on tensor board before but one of the reasons I'm doing one again is because the embedding visualizer is a pretty cool tool and I'd love to talk about that and it's a new tool and it's going to be really cool so in this demo what we're going to do is we are going to look at MN ist now I know you guys have seen Emma nice a million-in-one times but the point of this is not immunized T it's the embedding visualizer specifically so that's what we're going to look at and so yeah I'm going to start off by showing the code that we're going to use and then we're going to visualize it in tensor board and then we're going to look at the details for each of the sessions alright so let me start off by looking at the code here so what I'm going to do is I'm going to explain the code and then we're going to go into the the session itself okay or the code itself alright so to start off with let me remove some things so to start off with we're going to look at this code so I've got it up here I want you guys to download the code and follow along with me okay so what this code does is it is a convolutional network it's a convolutional net and it is meant to classify hand written characters okay so let's look at what it is and let me say let me start off with a two minute QA okay so ask go ahead and ask questions and yeah it's gonna be awesome we got code we got questions we got sublime we got audio lag yeah all the things were here we are all tensors okay any questions guys before I get started because I'm about to go in go in on this no pun intended okay how do we okay how do we decide the topology of a neural network great question so many decisions go into that depending on your use case what you're trying to do but I would say there are a few general rules to go by rules of thumb one would be the more layers the more computation but also the more accuracy your model will have a what type of network it depends on your use case convolutional network for images recurrent network for time series data feed-forward network for binary data and then more complex data there are different types of models two more questions and then we're going to get started audio is a little low we'll take care of that how to use tensor board with Floyd Hub uh I haven't heard of Floyd hub but I actually I've heard the term but I don't I don't know what Floyd hub is uh how much product goes into that hair so surprisingly none it's just I just roll out of bed and I can die and I do this one more question not about my hair how do we use tensor board with AWS servers now that's a great question so I actually have not seen that done before uh using tensor board in the cloud would be a good idea if you had a if you were using transfer learning so if you're using a model that you yourself hadn't trained and it's a huge model then using it without having to download it directly from the web would be actually that'd be a great startup idea so not just AWS but just visual so here's a great startup idea for you guys visualizing a modeled architecture in the web is a problem that has not been tackled at all right now we have to do this locally and no one wants to have to do that so great that would be a great problem okay so let's go into the code okay guys so let's so what I'm going to do is I'm not going to type out the code I'm going to talk about it and then we're going to look at the tensor board then we're going to look at the tensor board version okay so to start off with let me let me maximize this code so we have a good look at it okay let's maximize so here are libraries okay and then for URL live I have this you know little versioning because you know for Python 3 versus Python 2 URL live is annoying so it's we have to have this and so then these two constants these constant values are what we are doing here and so this is the this is the novel part is we are importing the we are importing the embeddings from the web so let's look at what this link is so on github if we look at this link we can see that it's an invalid request that's what we can see but there's also let's see mmm so okay hold on okay so here's what we're pulling we are pulling these so what are these so let me I actually have these locally so let me just open them locally so there are two things here that we are pulling the first is a collection of sprites so these are sprites so what are these so what we are doing is we are classifying handwritten characters now most of us are familiar with this with this example but this is novel so for the embedding visualizer which is a new feature of tensor board we can visualize the learned vectors that our model creates when training on our input data and input labels so we have characters right 1 through 9 and these are what are considered sprites so when we learn these mappings from our model when our model is training its learning what a7 is it's learning what the and these vectors represent the correlation between the actual number the the number 7 and the image of 7 and that correlation is represented as a vector so the way that we can then represent that vector is to have a sprite and these are the sprites and so this is this this image art is going to be cut into cut into squares okay so starting from the top left to the bottom right we can think of it as a matrix and when we visualize it in tensor board when we visualize that in tensor board these vectors are these sprites are going to show up as the as the images for what we've just visualized okay so just like this I'm doing this because I haven't actually launched tensor board yet but it would look like this right so see these fours and the zeroes those those are the sprites those are those are values that are coming directly from this sprite file and then associated with that we also have a we have a let me see where would I put this a labels file which is a TSV file okay so the labels file is a it's a file extension it's a file extension for a tab-delimited it's like for spreadsheets so it's imported into and exported from spreadsheets off and these are just the The Associated numbers these are the Associated numbers it's such a weird format but it every single number associates with one of these images from this matrix right here okay so each number is associated so that's what that's what we're doing okay okay so that's that's what that is let's keep going here so the next part is for us to download these embedding so what what this line does is it it is downloading the M NISC data and it's splitting it into three parts you have fifty five thousand data points of training data ten thousand points of testing data and then five thousand points of validation data and so we're going to call the images X and we're going to call the labels Y and each image is 28 by 28 pixels okay so that these are each of these images and that's what the read data sets function is doing having downloaded that from the web okay and and we're going to store that in M is T the the variable okay so let me increase the font size a little bit okay so now okay so now that we've done that we're going to define our layers so we have convolutional layers okay so up for our convolutional layers let's talk about what's happening here so in a standard convolutional Network we have what we have three layers and I'm going to answer questions in 20 minute intervals so in seven minutes I'll start answering questions but let's look at this architecture right here so I think an inception is a good example convolutional net right so in standard convolutional networks we have a what are considered blocks okay so it doesn't matter what these words say but we can just think of these as blocks okay so this is actually a huge convolutional net probably one of the biggest in the world in the world but the point is that we have a convolutional layer and in each of these blocks we have three we have a we have an input so typically a CNN is composed of a stack of convolutional modules that perform feature extraction so each of those modules consist of a pooling layer it's got end up and then a fully fully connected layer and then a what's the next one and then a softmax activation function so we squash it so we just keep doing that every time okay so we're going to define that programmatically here so let's go ahead and do that so so the first line here is the name scope so if so in tensor board we have name scopes right so name scopes basically define so this this one's not working so up right now but we're going to look at tensor board in a second but right now let's look at the name scope feature so there are so many variables there are so many tensors there's so many components to a computation graph so what name scopes do is there a way for us to encapsulate all of that complexity under one name so that it's easier for us to view in the graph okay it's easier for us to view in the graph so in this case we're using a name scope and we're going to define it when we call this dysfunction but the the name scope itself is going to encapsulate the weights and biases and the the convolutional part the activation part and then the pooling part of this this this convolutional we can call a block okay so a convolutional block consisting of these three layers a pooling of of an activation function and the weights and biases so this it's all going to be encapsulated and we're going to see that in tensor board when we when we visualize it in a second okay so so a PF variable maintains the state of the graph okay so that's what we were using variable for and truncated normal is going to output the random values from a truncated normal distribution and so then the constant is going to create a constant tensor okay so that those are our weights and our bias sees this and this is going to be for our first convolutional block okay and then we're going to compute a 2d convolution given a 4d input that is the 4d tensor that is coming directly from the placeholder that we're going to define later so right when we input data into our model it's going to go right into this convolutional block okay and then once we have that we're going to create were called histograms so let's talk about TF summaries for a second so what are these summaries and why do we use them so a summary is a tensorflow operation that outputs protocol buffers so protocol buffers are a way of encode are a way of encoding of data so it's it's it's serializing the data that we have in memory to disk so we're writing it to disks so that we can then pull it in tensor board and visualize it so that's what the TF summary function does and there are different things that we can use summaries for so in tensor board we have we have different types of summaries and we can look we can see what types they are by looking at the tabs up here so we have summaries for scalars for images audios graphs distributions histograms and embeddings so we have different summaries for both and we'll talk about when we use those summaries and why we use them but for this case we're going to use histogram summaries because we want to see a distribution of values across the weights biases and activations okay because we're randomly initializing these values right now right with this truncated normal function and as as we update those values over time there's going to be a distribution of possibilities and we want to show what they actually are and then also what they could be so then that and that's useful for debugging so we could then rerun our model with a different set of hyper parameters and see how those distributions move so that's what this function is and then we have a max pooling up we have a max pooling a layer which pooling in general is really cool so Pooley I mean pooling there are so many different types of pooling methods that we could use and max pooling seems to be the one that's used most often so if we could so for max pooling let's say we have a four by four matrix right we have a four by four matrix and that is our input image and we have a two by two filter so the reason it's called convolutional is because we are taking a filter which is kind of like a flashlight and we are conveying around an image and we're only using the parts of it that we find relevant so it's like it's like taking a flashlight to an image and looking at what those relevant features are and they're also these there's a there's another parameter called strides which are essentially intervals so at what interval do we want to look at the part of the image we're looking at every two pixels every four pixels every eight pixels and depending on what your stride length is it's gonna it know that we need to tune that so that our model is a better or worse and we can do that through trial and error and what what pooling is is it says let's um let's take the max of a region and just use that as the input so if we had a so if we had a let's see let me show a little image for that and then we'll keep going with this so this is pooling right so this is if we could look at our image because images are all matrices right why we use pooling so images are matrices an image is a matrix of pixels so if you think of this image right over I want to see if I can there we go so this image over here just that the big image over there and then we that that's our image so what pooling is it is saying what portion of the image do we want to use when we take that data and pass it forward through the network and we're with max pooling we're going to say well if we were to split these up into squares like four different squares if we were to add up the value in each of these squares which um which squit which big square would contain the most values and that's the max values and that's the school that's the square the subset that we're going to use and that's the subset we're going to pass forward in the network okay so and pooling is the most popular of the of the max pooling is the most popular of the pooling methods okay so that's what we did for one of our layers and right so if following along we established our weights and biases we are and then our convolutional layer we applied a we applied an activation function to it and we used riilu and so why do we use real ooh because real ooh reduces the likelihood of the vanishing gradient which it recall for for neural networks recall for four LS PMS and time series data the vanishing gray is a huge problem and it's also a problem in convolutional networks okay so that's that layer now now we have another layer and then we're going to get into actually building our model so this is our fully connected layer and so what this does is it performs classification and we use this in all sorts of networks it's not just a completely connected layers are used in almost every type of neural network and we use them pretty much before we use them before we squash it into using an activation function to output a prediction so fully connected layers are usually found at the end of neural networks at the very end and what they do is they and the reason that we use fully connected layers is so that we can use all parts of the data because we're about to because we because we want to squash it ok so that's what that is and we're going to do the same thing we're going to use we're going to use the histograms to create summaries for the weights biases and activations ok so that is that and so now let's build our model okay so we'll establish our graph and initialize our session and now we're going to have placeholders that and these are going to be the gateways for data so imagine we haven't done anything I've just all I've done here is I have talked about what those layers are going to be and then now we're going to actually initialize them so let's see if we have any questions and then we're going to talk about what this looks like how does max pooling work with and an image of only ones and zeroes oh that's a good question so the ones would then the the the part of the matrix that has the most ones would then be the pool that we use that's the max value two more questions and then we're going to keep going with the model how did he get this keyword okay if we had infinite resources we wouldn't do pooling right uh that's a good question so yeah yes the answer is yes yeah I've never thought about it that way yeah because all of a lot of machine learning is a trade-off between computational complexity and brevity brevity be that encode or brevity be that in training time like we want to minimize a training time minimize the amount of code we have minimize the computational complexity so that we can maximize our and but also maximize our prediction accuracy so one more question and then we're going to get started with this does max pooling use a sig Monde a sigmoid function no it does not it does not max pooling isn't a cig isn't that a sigmoid is an activation function that's not max pooling ism is an operation okay so now to build our model we have our placeholders for our images so that's going to be our tensors and then we're going to use this summary remember the summary function of tensorflow to create a summary protocol buffer for images so we can visualize these under the Images tab right here we're going to visualize that in the Images tab and then we're going to define our model so depending on how many layers we want well we have an if-else statement so if we have two layers then we can use that function we just defined - then - then I call the layer and then so if we have two layers then we're going to use that function we call it to build a model else we'll just use one layer and then once we have those layers we're going to at the end we're going to flatten it and then we're gonna feed it to a fully connected layer and we flatten it because it's it's more computationally efficient for our model to read a 1 dimensional tensor than a to demand two dimensional tensor however there are ways of not having to flatten an image and then using that two dimensional tensor directly but however it's more computationally efficient complex not efficient okay so those are our fully connected layers and I'm sorry those are our convolutional layers and then we'll take the flattened image and feed it to our fully connected layer and then we will create those embeddings so that we can visualize them later so at the once we have those fully connected layers that's where we're taking our embeddings to visualize so that's the part where the embedding visualization is going to is going to happen so data is flowing through our convolutional layers okay through the max pooling through the activation functions it's getting flattened and then fed into the fully connected layer at the end and that the result the result that the fully connected layer creates that embedding is what we're going to visualize and it's gonna be so dope when we visualize it it's so cool okay so and I'll answer questions in in five minutes so that's what we have for that and so and then we're going to take the log ins which is the probabilities that don't equal one it's it's the output it's we have the embedding we store that in the embedding variable and now the log gets are going to be the output of the fully connected layer hey Alexa hey computer do you like tensorflow no I was a little we had Alexa here but she didn't work anyway I thought it would be fun so anyway so the next step is for us to define our loss function so we're going to use the standard cross entropy a loss function with log 'its so we the log 'its are all those values that we output we squash it and then that's it once we have that we can then generate our prediction from it so this output is going to be our set of prediction values okay from that from this named scope which we call X ent which is short for cross entropy loss and the output of this so X nth is going to be our loss and we're going to visualize this loss in tensor board as well so remember named scopes are going to group operations together so that we can then visualize them as a whole in tensor board okay so then okay so now we've defined our model and now we can train it so we don't just define named scopes for our model we also want to define it for our for our tensors are so for our training operation and our testing operation so for our training operation we're using atom and so I've written down a huge explanation here of like why to use atom and when not to but in a nutshell atom is is gives us better results than the standard gradient descent optimizer but it's more computationally complex so if you're willing to make that trade-off then I would go for atom and most of the papers these days that are using convolutional nets I tend to see atom used more often than the standard gradient descent optimizer but if you're a beginner then I would go for gradient descent optimizer okay okay so and let me try Alexa one more time you think I should hey Alexa or hey hey computer what day is it today computer what day is it today it's Wednesday April 5th oh it doesn't like me okay the voice okay so anyway that works anyway mmm it's we have our training River accuracy hope so hold on guys we're gonna we're about to visualize this in tensor board I know you know I've been waiting for that too let's let's just get through this code okay so we have 380 people here from all over the world okay we are growing so fast it's amazing it is amazing so I am so honored to be here with you guys so where were we we have defined our training function and now we're going to define the the last name scope is the last name scope that we're defining here which is the accuracy so we're going to take the log it's and then we're going to which is a collection of the probabilities and we're going to choose the the average which using the argument or sorry not the app not sorry we're going to use our max which is going to get the largest value and then we're going to get the average using the reduced main function and accuracy is going to be one scalar value that is our prediction right there the accuracy okay my desktop is pretty crowded it's a lot of editing but anyway where was I make this bigger okay so now is it's time to look at the novel stuff about tensor board here so in a lot of the initial tensor board tutorials it didn't have this because the embedding visualizer didn't exist but now it does so what we're going to do is we're going to we the reason we summarize the all the some the reason we merge the summaries before we write to disk is because it's more computationally efficient instead of continuously writing them to disk we just merge them all and so it's one write versus like ten okay okay so where was I so we did that and now we're going to initialize our embedding matrix as an array of zeros and then we're going to assign it the embedding that we just calculated all right so now it's in our assignment okay so now it's in our assignment and now we're going to initialize a saver and the saver is used to save and restore all of our variables and so then we're going to use initialize a file writer so we have our summaries and then we have the file writer so the summary operation creates the protocol buffers so that we can write it to disk and the file writer is that it is that method that we are able to actually write to disk so we create summaries and then we save them to disk using the file writer so they both they both go hand in hand okay so that's why we use the file writer and so this part needs to be better let me just say this part needs to be better so in the initial tensorflow example so there there actually are not a lot of examples of tensor board that use um that use this there that you can look at the embedding visualizer in there not a lot of examples of tensor board that you can use the embedding visualizer in so there should be so this is one of them so definitely check out the github repo because there are not a lot of them the problem with this is that it's using the config file directly from tensorflow now i would like it so that we can define our own configuration file but if we look at this proto file let's look at what this looks like it's pulling it from the web and so let's look at what this looks like 404 here man we got it we got to clean this this up no it's not this there it is so here it is so that this is these are the these are the options that we're using to build our embedding visualizer so we have a tensor name we have metadata and then we have our project configuration okay so we could also define these locally if we wanted to by creating a proto a protobuf file locally and then calling it from disk but right now it's calling it from the web right here okay so we're good add multiple embeddings but we're just going to add one and that's what we've learned from our embedding matrix okay so we have that and we're going to specify the width and the height of it with a single thumbnail okay and so they're 28 by 28 pixels okay so that's it for that and then we train it and then we save the checkpoint every 500 iterations so this thing is going to run for two thousand iterations with uh with and each batch is going to be a hundred and then yeah and so that's what the training step looks like and then we have our main function where we call it and then this a this make H per am strength function just converts a hyper parameter string to a one that is more detailed for us it's just for us okay so that's the code I wanted to kind of blow through it a little fast because I wanted to get to tensor board so before we get the tensor board let me ask if there are any questions before we get started with this okay I'll take two questions and this was working before I started oh my god you guys know how it is with demo is like literally it works perfectly it work you could bring run I literally stopped it and re ran it like three times before starting it boom tensor board ever you know that little command line with tensor board and it was running and then five literally almost like the gods of deep learning were of demo hell were there this they say they didn't make it go it didn't work so we're gonna figure it we're going to debug it in real time and let's see if we can make tensor board work in real time okay so we'll see so two questions how do you interpret the histograms I have trouble giving meaning to the axes ax that's a great question and I'm going to talk about that when we when we look at it are there any third party wrappers for tensorflow so as to convert symbolic programming to kind of simulation for end-user and object-oriented programming that is a big question with a lot of parts to it third party wrappers for tents v'lo yes there are tensor layer look at tensor layer Google that tensor layer are you using GPU version of tensorflow yes I am what is protobuf I'll do so protocol buffers Google invented this like a couple years ago like six seven years ago but basically protocol buffers are a serialization method so serializing as a way like pickling is a form of serialization it's a way of taking some data and converting it to some standard format like some generalized standard format so that you saw that so you can save it to disk and then recreate that data in a later form in another form later one more question and then we'll get started with this how would you detect fake news using ml and DL so guys we are at a point right now where the tools available to us to generates data are getting better and better so fake news will get more and more realistic but at the same time the classifiers we have to detect what's fake and what's real will get better as well so it's crazy if you think about it a viruses will get will learn to fight will get better at learning to find vulnerabilities and systems but at the same time virus detection algorithms will learn to get better at detecting what is trying to attack its system so it's this it's this battle is this constant battle with both sides getting stronger and stronger and machine learning is at the forefront of it and the way we make sure the good side wins is by spreading AI so making sure everybody has access to it because if only if you have access to it then bad things can happen so learn AI will tell your friends about it spread AI awareness tell everybody about it we've got to get this power distributed to everybody to prevent bad things from happening so we're moving into a very beautiful world and we have to make sure that everybody has access to power anyway so that's it for my rent on that now what we're going to do is we're going to visualize what we've just written and we're going to make sure that it works so what happened here is to run this so let's see okay so we want to run this in tensor board what have we done here we saved it to this log directory right here so it's in TM TM pmn IST tutorial and that's where we saved it okay that's where our file writer saved it so now what we want to do is we want to we want to train our model right so let's train this thing so we'll say make it bigger python where my MN is sita pi and that's going to train the model hopefully it will train it so good so good so good okay good so great so now it's training the model okay so while it's training and I encourage you guys to train it it's going to take about five to seven minutes on a CPU and it's going to take about five to seven minutes on a standard CPU on your laptop so don't even worry about not training it locally you could totally train locally so while it's training we're going to run tensor boards so to run tensor board now all of you who have tensor flow will have tensor board there's nothing extra you have to do so let's initialize center board I can do it I can do a Q&A a livestream someday but right now we're going to just run tensor boards so to run tensor board we run tensor board and then it's what was it it's log directory equals and then the path that we saved the logs to so what happened is it stopped working five minutes before the demo so I'm going to paste it and let's see what happened we're going to debug this together it's definitely gonna throw an error for some reason which is so annoying but it's it is what it is come on baby let's do this okay so this is what I'm talking about so normally this would show the URL that we could then in our browser to visualize our tensorflow computation graph but instead it's showing this error tensorflow starting tension board be 41 on port 606 so I don't know what that is and so I was googling this one minute before the stream started but I found that someone else had this issue tensor board isn't showing and this was a not a recent issue this was a year ago but he had the same issue and then Dan named who I interviewed who is also the guy who's you know in charge of this this stuff on the tension field team said something I'm like blah blah blah blah blah I mean this this thing goes on forever but what's the result so this what I do I just kind of like zoomed through the this the stuff and so okay so he had debug so he so maybe adding a debug flag at the end would help so let's see what happens here okay and guys if it doesn't work we're gonna we're going to visualize it somehow what we'll figure it out okay so debug let's see oh no [ __ ] there's a shot right there okay you need this one no it's okay okay so then we could try it again with the debug flag as the github issue says and okay so yeah Ben has good idea let's just open up localhost and see what happens localhost six it was in six or six no it was six six six oh six I'm Lee pause training here because it's really taking up compute and I don't need that right now okay so so something here is loading up okay yes okay yes I'm so happy that this is working hey guys I'm so happy that this is working right now this is the greatest thing who is this god Ben Benjamin Shultz Larsen shout out to you wizard of the week wizard of the live session okay so here's what's up let's visualize this okay so um where were we for our scalars so let's look at the you know what let's just go straight into the embedding because that's what I'm most excited to look at look at this you guys who have stuck around have yet to see this amazing amazing amazing visualization so let's talk about what this is so what this is is it is our these are our embeddings that we generator to remember in our fully connected layer we we fed those into our embedding visualizer okay and the way that we are visualizing them is with those sprites so remember the sprites are these are these files and it's like a matrix we're cutting it it's like a matrix of I don't know how many it are there there's like 28 by 28 or something like that but these sprites represent each of these embeddings that we've learned okay so what what we're looking at right right now is a visualization of this graft and we're using PCA principle component analysis as a technique to map them out so so let's talk about what this what this is okay so PCA versus t-sne is is a good question so when would we use PCA and when would we use pts any so um actually I had my notes on that so I forgot about that thanks so PCA is a technique that we use to visualize data and it's actually most of the time you'd want to use PCA over t-sne most of the time okay most of the time we want to use PCA over t-sne but in this case but and let's look at what t-sne looks like boom let's find the nearest neighbors using t-sne and watch it move let me let me make this big no wrong way the hell okay okay so you see that they're clustering here these values are clustering in acids where they're supposed to be so all the sixes and all the threes and all the fours they're all clustering together and what we want to do is we want to compute the distances between them so there's a lot of things that we can do here once we have them visualize and we're going to talk about what all those things are that we can do once they're visualized okay so before so there's a lot we can do here and the idea of SNe and t-sne is to place neighbors close to each other so that's what this is doing and but the thing is that it almost completely ignores the global structure but PCA is the opposite it tries to preserve the global properties and those are the eigenvectors with high variance while it may lose a it could lose the low variance deviations between the neighbors so it's a trade-off and there's actually a great Stack Overflow link for that in the github like right at the very top the stock exchange showing like five reasons you want to use PCA over t-sne so that's what that is so what we can do here is we don't even just have to visualize the we don't have to just visualize the okay so there's a lot of comments here about my hold on my name and stuff how many people we have here live yeah we have four so I'm here to say this to four hundred forty people live okay my name is Suraj Raval I am India my parents are from India let me just say this okay when I was 18 years old I legally changed my name to Jason Scott Ravel's when I was 18 years old because I wanted to do great things and I felt like the only way to do that was to be white or at least anglicized it was only three years later that I legally changed my name back to Suraj Raval I learned to love myself over time I learned to love my identity and now I am unabashedly Siraj Raval an Indian and an American so I just wanted to say that publicly it's embarrassing to admit but it gives vulnerability but it's also a point of it just needs to be said it needs to be said to the world I am Suraj yes my parents room India and I love being Who I am okay so I am very Indian and I love it love Indian food love Alda Indian culture love India visited went there for six months and yeah I just wanted to say that life so where were we yeah a lot of racism growing up in Texas but anyway back to this welcome back to PCA and t-sne where were we where were we so what are we doing here dimensions so we have to Deaver tsa's 3d so we want to there's there's several things that we can do here with our graph and there's a lot of things happening at once let me focus here okay um thanks guys I appreciate the support anyway I guess I did want to see the comments and see what people thought you know it's one of those things that I haven't really admitted before to a lot of people but you guys are my you guys are my homies you guys to my crew and I'm just going to continue being real with you guys I'm you know I'm good I'm going to continue giving you all of me all of Suraj every part of me every part that I've always been afraid to show whether it be rapping about t-sne or whatever else it is you have a very unique part about you and you probably don't even realize this you have a very special skill set and if you find a way to encompass all of those things you can do in 2-1-1 activity you will find success and that's what I'm here for I'm here to help you be successful that is my goal that is the reason that I do this I want to help you become awesome because if you're awesome then our society will be awesome this is power unlike anything we've ever seen before if you're able to understand this stuff you're gonna be amazing okay you're gonna do great things so where were we haha Suraj you deserve a plate of biryani that's my favorite comment of the day we're worried I would love some okay so um man I'm just saying all sorts of things right now we were somewhere else we were somewhere else right now we were uh we were talking about t-sne so we have our tensors and we have our the color map in our labels and our dimensions oh you can change the colors too so check this out you can change the colors too so we can change a color by label we can us so when we spear eyes the data it it puts it into it more ball-like structure and we can also search for different things that we want so if I can say seven it's going to show all the sevens so this can also be used for word vectors so remember in one of our videos we talked about the difference between man and woman like man plus woman equals well no man plus woman equals child or something like that but can Queen that also works in here and I would love to show you guys that later but natural language processing later on so yes so six five four three two one that wasn't countdown to nothing I was just typing this out how cool is this though you could you could you could visualize it in and you don't have to just visualize your data you can also visualize the weights you can visualize your biases you can visualize a lot of different things and in fact you can visualize stock prices what I would like to see and it this depends on the tensor flow team releasing plug-ins for tensor board is um is uh I would like to see more plug-ins for this so I'd like to see more people make things for this so we can choose different compiled what else can we do we can do o'night view that's that's that's my stuff right there and we can also select parts of it so look at this so if we select this bounding box selection and we select some specific part of it well we could then get those parts and then I want to I want to show the cosine distance so let me show how to do that so if we were to isolate the points that we've just bounded in a box we could then find the cosine similarity between those points which is so by the way bookmarks are to share this with other people so you can download it and share it with other people now I want to yeah sure so thanks and we can do custom visualizations as well we can say like from 3 to take these labels and then match them using some vector use case a random vector anyway let's talk about the other parts as well there because it's not just this there's more to it than this we have 9 minutes to go right is that okay we have 9 minutes to go those the last 4 minutes are going to be for questions let's talk about these other parts so the scalars tab is for scalar summaries those are single number values that change over time ok and what we have here are the accuracy and the cost function which is X ent the x-axis shows the time steps and the y-axis shows the accuracy or loss and if we can increase the graph for a closer look by doing this just like that and or to view a wider range of data points depending on the end which expands the y axis just like that ok we can also double click to zoom out so double clicking zooms out so we zoom in and then double click the zoom out ok and the step option shows the time steps okay and so a great thing about this is so right now we only have two of these two scalars but if we had multiple scalars so if our computation graph was really complex then we could group them together so if I type in accuracy or sorry if I was having like lol it's like what's going to be here nothing but if I type in accuracy it then encapsulate the accuracy that we already have so if we have names like you know accuracy slash accuracy one accuracy slash accuracy two accuracy accuracy three and then I were to create accuracy right here it would then encapsulate all of those ok so that's great for I'm doing in the browser encapsulation similar to name scopes but kind of like a GUI version of that ok save your question to the end and then smoothing makes the graph smoother or less smooth less smooth is if you want more the trade-off here is less pretty but more accurate what else we got we got relative and then wall relative shows the time relative to when it started whereas wall shows the time of training in general like in real world time what else we have we have our images so these are the the image summaries that we created we can visualize them right here okay and then we have audio so I haven't actually seen somebody use audio for tensor board and believe me I have looked I have looked on the web and I haven't seen anybody do it so I would love to see it I would love to see someone and I actually you know I'd love to see someone use wave net or one of these things that used to look at audio I think magenta might but magenta is so hard to get started there's so many scripts and it's very confusing so let's look at our graph I can't believe I saved a graph for last but the thing here is the graph is these are our named scopes right so we commnets to com1 fc-1 these are the named scopes that we created and if we double click we can then see the parts that are encapsulated in the named scopes so why are these colored different colors so for every color except for gray those are the default colors so for every color 10 circle automatically looks at the the data inside of these operations and if they're the exact same then I'll color it the same so it's kind of a it's it's a way of organizing data so it's cleaner and it's a way for us to know that it's going to be the same type of data okay anything I did this is because in it's an inception there was some bug and it was because in one of the layers it was like one small difference so now they have this by default so colors it's going to color code itself and if you look over here at the side you see this in it all by itself that's because tensorflow detected sorry tensor born that's our tensor board detected that there was one operation that was continuously used a lot and so then it put it into its own section because if we didn't then it would be it would look even it would look crazy so let me show you this we could put this in its function back into the graph and then it looks like this because it's used all over the place but we if we removed from the graph we could see all of its connections and have the graph be cleaner so the reason it does is this auxiliary auxiliary node function is to make it cleaner to view okay and then we can also structure it by the by device whether that's CPU or GPU or the structure and we can look at everything that's inside of the graph like this and all the layers increasingly so much complexity I love it I love it man I wish I had time to just make my own project and just make a huge just really go dive dive in on this but I'm having so much fun making content right now so I'm just going to keep going with this two more things I want to show really fast are the distributions and the histograms so the distributions are for their weights and our biases they're for layers remember we randomly initialize them but they change over time and we want to show what they actually are compared to what they possibly could be so that so the possibilities versus the actuality or reality and one more is are the histograms so histograms are for a histogram plot allows you to plot variables from your graph so if your model has weights the histogram shows you the values of those weights and how they change with training so this is not is this nice little 3d looking graph which which is pretty cool and yeah so that's it for this we've got three more minutes for questions so let me let me answer some questions and then we're out of here uh wow that so MA there's even more people in here now than the word before which is insane now there are four hundred thirty two people here okay so let me see what people the video did not pause in deep neural networks what is a what is an alternative for gradient descent for minimizing errors and deep don't forg written for minimizing errors so that's a great question there's stochastic gradient descent which is different from standard gradient descent there are a bunch of different optimizers at a grad Adam if you look on the tensor field documentation for optimizers you'll find a list of them two more are two more questions will deep learning solve intelligence so spoiler alert I don't think so I don't think deep learning will solve intelligence I think it's a pathway to get there we need to make more computationally efficient models with that need less data for the same result and I feel like there's still something that we're just missing something something very fundamental something very basic and yeah but we'll get there we'll get there uh let me know two more questions because we have two more minutes how do you explain the evolution of the weights distribution given the evolution of the gradients distribution so recall and I have a great video on this back propagation in five minutes that the gradients give us a direction to update our weights so they are and your question was it's gone now but how do we something about relating the gray into the weights so the weights a great way to model that is to have histograms for both the gradients and the weights right now we only have histograms for the weights so you can see how they both change in real time one more question two more does our brain use back prop no it does not our brain does not use back prop uh Adam optimizer versus SGD I would say SGD and one last question is l-bfgs a true linear regression technique uh l-bfgs is one of the most complicated things that i've come by I have to admit um I I don't think it's a linear regression technique yeah anyway yeah bless your folds l-bfgs was blesser fetcher guest your session or something like that it was the name of four scientist but anyway so that's it for this live scream we got 400 people at the Animalize from that's amazing thanks guys for watching for now I've got to go make more technically accurate more mathematically accurate content for you guys because I really want to be increase the learning capability of my content so I love you guys so thanks for watching