Balancing RNN sequence data - Deep Learning w/ Python, TensorFlow and Keras p.10

Key Takeaways

what is going on everybody and welcome to another deep learning with Python tensorflow and Cara's tutorial video in this video we're gonna continue building on our a future crypto currency price movement predictor recurrent neural network okay so where we left off we've got this pre-processing happening we've built the sequential data and we've separated out our validation data we have we normalized the data here we scaled the data here the next thing we need to do is balance the data we need to have as many byes as we have cells if you have any more like if you're like 52% buys and 50 want 52% buys and let's say 48 percent cells let's make sure it adds up to a hundred if you're if you have data like that it's probably ok you probably need to worry about balancing it but if you've got anything more than like a 60/40 split or even more egregious it's really important you balance your data set and arguably it's just always gonna be better if you just balance it because the the model is gonna learn probably pretty quickly if there's like a 510 percent Delta there that oh just always break this one class and immediately we make huge improvement and that's the easiest change that the model can make so it's important that you have a perfect split and imbalance of your data so that the model doesn't waste time doing that and then also kind of get stuck in a rut like it's gonna start doing that and then from there it might stay stuck and lost and so you don't want that to happen now there is a way you can pass class waits to care us and tell it like hey one example of this is worth 1.5 examples of this or something like that and you can tell it hey wait these a little different when you go to calculate the loss for mistakes made and stuff but to be honest I've not found that to solve balanced issues so I think that's a great idea in theory it just doesn't seem to work so anyway balance your data so so that's what we're gonna do now we've shelled the data and now what we want to do is actually you know balance it so what we're gonna do is and again there's probably better way to do this but I'm just gonna have two lists here we're gonna say buys as a list and sells is a list clear that out we're gonna zoom in a little bit okay then we're gonna say is for sequence targets in sequential data if Tarija is a zero that's a cell so then what we're gonna say is cells not append sequence target Elif target equals a1 bys append sequence and target okay um so now we have buys and sells the next thing we want to do is before we start slicing things and dicing things we're gonna go ahead and do a random shuffle buys and dot shuffle cells we probably don't really need to do that because we've already shuffled here but shuffle for good measure so so then what we're gonna find out is which one is lesser we don't actually need to know if it's by ourselves we just need to know what's the minimum value of these two lists so the way we can do that is lower equals the minimum of the Len of buys or the Len of cells and that'll tell us you know like maybe by cz's 30,000 in cells a 60,000 well lower would be 30,000 then what we can say is buys equals buys up to or I'm sorry let's see uh how do we do this up yeah up to lower so if the Len is 30k it would say up to 30,000 buys and then sells equals sells up to lower I'm gonna remove 30k because that's not true I don't think it's just a value okay okay so now we've got that so then we're gonna say sequential data now actually is just equal to byes plus the cells and then random dot shuffle sequential data why are we doing that shuffle again at least this one is so like the data isn't all byes than all cells cuz that'll really confuse the model you definitely want to shuffle it up okay so now we've got that what we need to do is now it's in sequential data we need to convert it from you know it's bison cells is containing the guitar you know the features and the labels what we want to do now is split those out to be you know the X's and Y's as diff you know as different lists I guess because the way that we're gonna feed it you know it's going to be a model dot fit XY so we have to split these out into X's and Y's so we're gonna say x equals a list y equals a list and then we're gonna iterate over the sequential data so full sequence targets in sequential data we're gonna say capital X can we type dot append the sequence and then Y dot append the targets okay at this point I think we're closing in on being complete we're going to return the number array of x and y so our pre-processing data frame function should be complete and now we're ready to do is come on down here and comment out that uncomment those get rid of this and I think what I want to do is let's print out some statistics I'm not sure I really want to type all this out I'm not going to I'm gonna copy and paste this from the text-based version of this tutorial this is just wastes Aria but my dog so long I think our trash people are here and they're making a bunch of noise oh no we got a huge I have no idea what's going on someone's putting a ladder up to my roof I probably should be somewhat concerned okay it's so the so what we can see is the training data is 69,000 validation is 3000 there's a good balance between the buys and the don't buys in that we have a balance between the validation buys and don't buys so everything is nicely balanced we've got it split apart the way that we're expecting to and in the next tutorial I think we are ready to build the model and train the model so that's what we're going to do in the next video if you've got questions comments concerns whatever you know the deal feel free to leave them below otherwise I will see you guys in the next video