Generative Python Transformer p.6 - Testing larger model

Key Takeaways

what is going on everybody welcome to part six of the generative python transformers videos in this video we're going to be just kind of testing and tinkering with the uh latest model that i have trained on a much larger data set for a much longer period of time now from what we've found you can train your own pretty quickly and easily but i'll also um host this model for anyone who's interested so i will at least host on nnfs.io so i don't know if anybody's heard of the website but uh also on that same website it turns out you can get access to the nnfs or neural networks from scratch book which is building neural networks from scratch in python essentially building your own framework but the reason isn't necessarily so you can write your own framework but instead to give you a much better understanding of the frameworks like tensorflow or pi torch that you might be using so anyways check that out if you're interested otherwise like i said i'll put the model in the description a link just so you can download it essentially it'll be this gp gpit1 directory inside of which we've got the tokenizer from the 75 wow you can't see that great start to the video um anyways it'll be this gpi t1 directory inside of which you've got the tokenizer from the 75 gig data set that i had and then also the latest model which contains all the model files as well as the optimizer so if you wanted to keep going uh with my training you could or if you wanted to fine tune and all that stuff so anyways that'll be there i also might put it up on hugging face at some point i don't know if i want to train it further or what i want to do i also don't know exactly how to put it up i just know that you can um so maybe i'll make that available as well so uh with that uh let's go ahead and do some tinkering so let me pull over some notes as well and what i'd like to do is just kind of like show the the model as it is uh in this state and then also kind of talk about like one of the many use cases but also kind of where to go from here so um so what i think we'll do is we'll just kind of modify this uh we don't need to do we're not training anymore we're not data collating we don't need this we really only need this first import uh we don't need this and uh i'm going to get rid of that input even too making quite a few deletes we kind of want this stuff below though so we'll get rid of this we do want that model i'm going to change the name in a moment and then we kind of want this stuff but we don't really want this for beam and output i'll just do this we'll say for now we'll give it a good old pass and i'm also going to uh let's go ahead and cut this we'll make a new uh put that down there what did i ru i don't think i didn't attempt to run that but okay i knew we weren't quite yet ready because we need to put the correct paths for both of these so that will be um what will that be for me is it going to be yeah so that'll be uh g pi t and then it's g pi t one right yeah uh and then for the tokenizer it was g pi t talk 75 gigs so you can pass that in for the tokenizer again you can use your own or use mine doesn't really matter then coming down here for the actual gpt2 model we're going to grab this way copy paste and then i just called it latest model um i'll try if i open like if i upload another one it'll be gpt like two and so on i might come up with a better way if i continue doing this for a long time anyway um and then yeah don't forget the two cuda and also let's see um yeah so i guess the only time the only other time we'll need the two cuda yeah is when we do the tokenize okay so let's go ahead and run that make sure that works and then if it does we will kind of check out the model as it is right now so uh let's go ahead and just make um imp ids imp so amp equals uh and then we'll just start with a real real softball in it um so the hope again here would be okay let's finish the init method by maybe some parentheses maybe a good old self maybe a nice colon and so on that's what we're that's what we're shooting for so we'll do that and then also let me make sure we are prepared for okay cool uh so that'll be the input we'll tokenize it here we'll generate this beam output and then for each of those in that output um well what do we want to do so in this case number return sequences um and in fact i don't even i'm not sure why i use the word beam here i don't even think that's act really accurate um i'm not sure if i copied that from somewhere or what because really it would be like model output and then it's not really the number of beams that are getting returned it's this number of sequences um so let's say three map unless i'm unless i'm mistaken right now i'm sure somebody's gonna yell at me for that previous video uh we'll call it model out um max length we'll set this to a hundred we don't need it that big and actually for a number of beams i'm just going to go smaller 10 beams is kind of a lot um so 4 we'll call it out in model out we'll print the out so this should be like token ids right so so again in this case the the the thing that we're going to iterate over is not beam in beam output it's um each of this each of the sequences now interestingly um this is like the more basic way to generate output but i will also show um you can like you can get if you wanted those like beam outputs and the scores and all this like really really deep stuff what you can do is modify this slightly so we want return is it like return underscore dict in generate we set that to true and then we also will do output scores true let me just make sure that's correct yep um so we can do that and then when we do this the output is actually going to be a dictionary with three things it's like sequences sequence scores i think and then uh like scores and then the scores are like per that is per uh generation per beam i think so each map anyway i'll explain it when we get to it so let's just uh let's just start with 4k in model out we'll just print um we'll print k and in fact um i might actually post this notebook so kind of what i think i might do is uh let's let me just copy this let's edit undo is that right i just want to leave that old method just in case we'll paste in the new method we'll run that again also if anybody's watching and you don't have a high-end gpu to like have trained a model on at this point you can totally follow along you can run this on a cpu uh at least on my cpu for example so on my gpu this runs in about depending on how many it's really a question of how many number of beams and then how many sequences you choose to return at the end but really it's a function of how many beams are how much how wide is that beam search going to go but in the case of like something like around this let's say it's about a little less than a second on the gpu and then it's about four seconds on the cpu so we'll just kind of generalize and say it should take you four times as long to inference but you can totally uh play with this model on a cpu it's a little hard to like train the model but if you just want to play with it do feel free you can you should be able to download this model and and do exactly what we're doing here or do other things or try your own um try your own um besides what we're doing here like with autocomplete you could try your own projects or if you do have a gpu you could also do the do the typical thing that you do with gpt where the p stands for pre-trained is like fine-tune it to some other tasks so anyway continue on so each of these keys the first thing is the sequences so up here these are your sequences so if we said um uh for um i don't know out in model out sequences print out in this case you'll see they are the same thing as we had up here essentially so you can still access the same data it's just formatted slightly differently and the reason why we're going to format those slightly differently is so that we could in theory if you wanted to access both scores but also so you can there's two things you've got the sequence scores these and then you've got um the uh scores and that would be like per mask so the first order of business i think we'll do now is we need some way of i think i'll just do it like right above here because we'll continue working here we're going to quickly define the new line character that will be the new line tag that we defined and then we're going to define a couple of quick functions here we're going to say uh encode new lines that will take some sort of input and all it's going to do is return imp dot replace and we are going to replace uh new line characters with the new line care that we set above there and then finally uh we just want to do like the opposite of that so rather than encode we're going to decode and then we take that input and then basically we'll do the newline character here cut we'll just paste it right here and yeah so from here what do we want to do so let's take this imp and well in this case we don't really have any new line uh but we're we're soon we're going to one we're going to want to be able to decode uh the sequences immediately uh and then soon we'd like to pass a little more than just duff in it but for now i guess we'll just use the d code in this at this point so uh let's i guess we'll leave this k um let's go ahead i think i'll i'll leave that and then we'll just continue working down here so so now we'll say for sequence in model out sequences what we'd like to do is print um tokenizer tokenizer dot decode and then it's whatever that uh sequence is so we can print that out real quick and you can see it's like really hard to read this obviously we've kind of already seen this before so then we're just going to decode the new lines so we'll just wrap this in that decode new lines and now it's a little more legible and one thing we can do is i'm just going to copy this over to somewhere where we will see some highlighting boop and as we can see i mean it looks pretty good obviously there's this again we had the max lens set to 100 so i think it just ran out of length um otherwise these are looking pretty good there's some subtle differences here uh interestingly um for example uh here you've got def in it and then you've got you know tabs over for the dock string and then after the dock string it double tabs over not really sure exactly why it's doing that and then here you've got the def in it and then double tab and then it continues with this double tab so at least this one i can sort of make sense of because even though we started with this the model after a def in it probably the most common way the model has ever seen you know subsequent stuff after the new line after you have def in it is with two tabs because and why would that be right it would be because the first thing was a class you know something with a capital s of course uh and then this was all going to be tabbed over and then when you got to the new line it would have been double tabbed as it was so um so that kind of makes sense to me why that happened and then here it actually everything was tabbed all the way over and again this is just based on confidence right so even though the model did present this to us um it was less confident about this one than it was for this one and then even this one so something to keep in mind my guess is it's really the model is just confused because we've got this this clear init method that is all the way left oriented so anyway probably not the best test based on these results but moving on so we can seek model out sequences and then the other thing that we can do is so for example so we've got three that three comes from not the number of beams so if we set this to for example five and run all this again you'll see uh again we have three things so even though we had you know five beams going out and searching we're still only returning three sequences so the other thing we can do is we could say four uh uh we'll call it sex score in model out uh sequence scores we can just quickly print the sex score sequence scores what does do wrong sequence says scores okay sure so there you can see the scores um yeah they are negative you also probably if you wanted to really use them you could move them to cpu and so on but anyway that's how you can access this score for that's like the score for the full sequence but if you want to go even deeper you can um for example uh we can let's check out uh model out scores so let's print model out just scores and in fact let's print first the len of that okay so in this case you have 96 okay what is that that is for each of the generated tokens at each step of the way so it's a little less than 100 but if we change this to say 250 right because apparently def init is probably in total four tokens so this should be 2 46 so let's go ahead and run that run this blah blah blah blah come down here and then yeah so you get 246. so this what this is now so what if we um for thing in model out well actually four um for token for to i don't know what to call this for mask token i guess mask token in model out uh scores now print len mask token let me see um and in fact let's just do a quick break here in this case you get five so we're just going over the first because there's going to be 96 of these or 246 in fact so let's go back and fix that to be back to 100 i'd say for now and uh so we have five so where's this five well the five in this case is corresponding to the number of beams because you're doing that beam search so you're going to allow for the top five beams and again why would you do this the reason for this is you have a score for the entire sequence length but sometimes the best total sequence won't necessarily always be based on going with the highest confidence of each mask sometimes you want to go with the second or third or fourth or fifth highest and then the end result will be much better so the number of beams that's basically what you're doing is per mask at least as i understand it per mask you're going with the nth top confidence so the more that you're willing to fan out sometimes the more diverse results you might get and then some of those diverse results might actually be the best results so anyway coming back down here uh we get five of them because of the beam so four so here four beam in mask each of these individual mass tokens we can then again print lan beam and i believe in this case we'll see 52 000 yes so in this case again this is going to be a vector of all of our vocab which is 52 000 it's going to have the scores for every single possible token in our data set so anyway that's a that's kind of an interesting one there's so much cool stuff that could come from this point uh so just be aware of the existence that you can you can dig into this but at least for now uh the main thing that we're curious about is going to be uh this right here so so let's imagine um that we are going to just simply do autocomplete so in our case uh let's just take uh we'll take this copy that we're just gonna i'm gonna do define auto complete based on some sort of input we're gonna run this imp is here cool and in fact the first thing we'll do is imp equals imp dot or imp uh what was it uh encode new lines imp that way this is encoded before it goes to our tokenizer for the tokenizers encoding so again this is just adding that replacing the new line character with the newline token we'll call it so that comes in does that we get our model output and i think probably what we want to do is um i think for now i'm gonna comment this out uh do that this um what i'd like to ask is print um in fact i mean it's not actually that long so i think we could probably leave this in um print um print imp dot count new line character oh we need to actually run this um imp uh we'll actually let's uh we'll leave this up here and then we'll come down here and we'll say um example input equals um uh let's just do um let's come up with like a really basic script so we'll say i think it's matplotlib inline is that acceptable matplotlib uh the single thing so first we're going to say import map plot lib.pyplot as plt we're going to import numpy as np and we're going to say x equals um range 5. i think we can do this y equals mp.random um mp.random um uh what would be an acceptable um np let's see mp.random is it normal yeah np random normal and then we'll say random this is probably not the way that people would do this uh normal and then we'll say size uh len of x plt dot uh plt.plot x y and then plt dot show i know that's off the screen there we go on that okay so this will be like our our sample test script um for for autocomplete so let's say uh you know so in this case as you type import matplotlib the hope would be it would auto complete as dot pi plot splt or you would import you know you do this and then import numpy and the hope would be that it would return as np and then as we get down here the hope would be could start predicting some of these things and then for sure after x and y are defined maybe a plt dot plot after a plt.plot probably a plt.show unless there's more things you know defined you know up above but something like this so um okay so what we would do um you know imagining this is our script we might say our input and in fact i think what i'll do is i'm going to cut this i'm going to come down here and start working on it we're going to say our input example input and then for now we'll just make it uh let's just do this and we'll just see how it goes from there so um uh how do i want to do this logic i guess i'm trying to decide how i really want to do this but i think what i would do is i would say auto complete so uh let's just do auto complete example input and then hopefully yeah so so the count of the number of new lines here is zero so that so we essentially want to divide up the output so we get the model out we're gonna get that um and in this case we'll just say sequence equals model model out and then it's sequences and then we just i think for now we'll just take the the first one so we'll say sequence um and then we need to decode that with our tokenizer so where is so it's just tokenizer.decode and then we we could decode the new lines first or it really doesn't it's kind of irrelevant when we do that step so um so come down here paste uh we will say decoded equals that decode new lines uh sequence cool let's go ahead and print decoded um what was our input okay oh wow wait that's crazy it's kind of weird that ah you forgot the as and then it gets it down here but then it automatically went and imported numpy as mp i think that's kind of interesting what if we do this what's noki it still does empire and it starts doing this okay okay whatever um decoded um i am kind of curious to always see the decoded but then we want to do the auto complete is going to be uh equals decoded dot split oh shoot how the heck are we going to do this i wish there needs to be a way and there might be like to split without removing the thing you're splitting by i wonder split without removing things split by python that might be like a parameter without removing the delimiter what's this doesn't split d no okay so the idea that i had is looks to be what you're gonna have to do okay whatever um so yeah that's weird that should be a function that should be like an extra parameter in split um probably is someone's probably madly typing below anyway autocomplete dakota.split um i kind of want to do this before the new line that way we can use the new line character it probably doesn't i don't know i don't think it matters anyway split by a new line so autocomplete split by new line and then what we want to do is split up a new line um let's say actually what i want to do is cut this we'll say splitted split equals that that's fine autocomplete will equal nothing and then what we'll do is 4 s in split 4s in split up to oh shoot new line count equals imp.count so we'll take this new line we'll come down here newline count plus one what are we going to do we're going to say auto complete autocomplete um auto complete uh what am i doing here plus equals s plus new line character man so even if you have like double new line i think that should still work because it would be like an empty string i believe in the split so so at the very end of this let's just return auto complete and uh yeah ac class autocomplete um let me do the following real quick print uh 20 times a dash uh and then we'll do the same thing down here i just want to make sure we have like plenty of space because i always kind of want to see the decoded part um like the full but i also want to see yeah this okay cool so we never printed the autocomplete so we'll go ahead and now also print ac do we already what have we done here um split for s split d code split by new line okay oh you're so oh goodness okay so this is an index not a slice um so we're saying hey new line count which was zero this is saying plus one so the first is so this is for every letter and not this one but this one so okay so yeah welcome to um welcome to python um so colon uh up to this number cool okay so now i will remove that and we are finally to our point where import matplotlib we say autocomplete equals whatever you know autocomplete this input and this was just kind of our debug information this is the true uh output and it said as plt that's kind of interesting i thought we were screwing oh i i thought it was screwing that one up for some reason but okay it appears it worked um i want to kind of clean up a little bit uh so i'll remove the split part um and then just to make things super clear auto completed cool we'll continue cleaning up slightly uh i think we need to do that okay so this is like the full um thing but just the auto completed import map lib.popular splt so that worked cool um so now we'll go import numpy so we'll just copy this and we'll make that the new um and in fact let's do this paste uh come back up here and uh copy that come down so we'll try this now so this one actually has like a new line in it okay so that one also um worked import numpy as np cool the rest of this again not really what we were after but how would the script know you know so it's just you know trying to figure out what the heck we're supposed to do here um but the autocomplete to the new line it worked cool uh what would happen if we said num would it know by that point no but then the next thing it imports is numpy is mp uh lame lame num points really num points is gonna be the suggestion rather than numpy it might be like that might be because of like how things are tokenized like clearly bro yeah so like if you just import the next suggestion like after so if we just if you inputted just this the next suggestion after that is to import numpy as mp which probably is the most common next import so that makes sense uh so i'm going to go ahead and continue so we'll just do another one so let's go to the next line where is my okay let's see what happens after we define x because that's not very informative my expectation is going to be that um if we at least define why that will be the first time that it could be helped but let's see what we get here oh my gosh that's a little odd that's not going to work out for you um oh no it does stuff here x y oh um outrange plus one i that's just i think we're gonna get that can't possibly work that's curious why would that be yeah like that i didn't think that was even logical like i didn't think you could do a plus one but otherwise it does work it's so close to finishing the rest of our script which is pretty cool um but no oh no did we what have i done oh this was the auto complete okay so um arrange five plus one come on brother don't do that what if we what if we enforce a new line what if we do that what range three come on man what was it going to continue doing for j for i and ranger yeah i kind of want to let this one keep going at the initial outset it doesn't i don't think that's going to work out um take me there oh no what a did you change oh here we go what in the heck are we looking at let's copy and paste it let's see what happens if we uh if we take i feel like copying and pasting on uh in a notebook is weird does that work no range object does not support dang it [Laughter] so it's not valid code it looked good though it was good enough to um to uh make me not certain if that was actually gonna work so anyway i'm gonna go back to the hundred characters just because i don't wanna go too too deep into things but anyway so the x to new line and then it at least defines why immediately again it has no idea that that would be the next thing it's just clearly has learned that so um what if we define why what if we just say y so i'm going to come down here uh we'll paste this in oh we we threw in a new line there yeah that that was of course gonna be weird okay gotta give it a fighting chance yeah it still wants to generate this weird you know it still wants to say range three right um that's not accurate that's not a good autocomplete uh y equals let's say y equals np we'll just start it off with something smarter okay okay all right okay this is looking good so so this that's good i think right because this will be anything from zero to one i think it's linearly and then there's five points along the way so this i'm giving it the sentex stamp i think this is actually gonna run so if we ran this it did it it did it oh man that's so exciting also again what's cool obviously it does like this stuff um this actually would still run this the whole the whole thing uh would indeed run i believe because this just comments out that and then if name equals main yeah so this is actually all valid that is so that's cool that's exciting i don't know i just man that makes me excited there's been a lot of kind of not so great although i guess most of the first line auto completes have been relatively acceptable um but this is really cool to see for me so anyway the actual so this was the full script if we just like let it go but again i think it's going to be very difficult to know like okay at what point does the generation start to get bad um i think that becomes very hard to determine so i think at least just letting it go to the end of the line um so far that those these have been relatively decent this is not exactly what we wanted but this is actually pretty impressive considering um like the five here it knows why they needed to be the same length because that was some of the mistakes that we've seen actually so far but anyway okay that's pretty cool uh we can continue down on this script as soon as i find it um i don't know random i'm not sure how much deeper we actually need to go and in fact i kind of want to redo this we'll come down here pasta come back up here um and obviously you know this stuff is you know this is just one example there's so many other things that we could be like doing and checking um random.rand um random.rand is it is that i can remember like random.random i don't know let's check it nope that's legit ah it's so fun it's so fun to me um okay so so how about one one more thing let's go to tensorflow.org and let me pull this over for you all um let's see where is learn i just just take me to the basic tutorial learn more see the tutorials uh beginner where's your mnist just take me to the amnest karas god i just show me some code okay cool yes um we kind of want this to uh yeah so let's do copy this we're gonna let's see if our model can make more models wait what just happened oh my gosh i'm getting lost i hope i didn't screw something up over here let's see i think it was this i think that was the what it was originally i don't i have no idea um yes i'm getting so lost in this notebook okay so uh in fact let's do let's just do this where are you gonna take us that's interesting um i take me further take me further where is our auto complete we are we're going we're going deep we're going to 512 now because i i need to see i need to see more of this so auto comp what did it auto complete did it just give us a new line and that's it yeah whatever um okay i don't know what this is um i also don't know what bn1 that's not really defined anywhere it was looking really good it got no good yeah okay one of the things that i did i did notice a long time ago was padding so as soon as you start seeing these padding characters in the generation you know it's gone bad so um that is kind of one thing i've i've learned um so like in this case the pad would be the next new line i don't know you would know as you got close to any padding to to watch out okay anyway um coming back down here let's just we'll just keep tinkering i kind of want to leave that one because that was actually kind of interesting to be honest with you um it does kind of like i bet it would import for example let's just see like what does it import from tensorflow.keras tensorflow.js import layers so it did got that right um okay so the next thing i want to do is what if what if we just do this so we're gonna copy this part from the tutorial so it just begins defining its sequential model let's see if it does that if it can handle that i might want to shorten the the length again but uh sequential we got down here tf keras so it starts off by making the hell happened here it closed it off and then it does activation rally that's not correct i don't think activation like that would need to be inside the conf yeah but i i'm not just biased it's false i don't okay whatever i'm sure that's a valid thing but anyway this is pretty i mean that's a oh and defined sequential model with three layers um it wasn't three layers that wasn't dense gives a summary it knows wow look at this this is so uh it just it makes me giddy and again when does this start getting messy actually i mean to be honest the pad is clearly being used as an unk token if you ask me um dot 2 device 2 dot 2 device okay yeah it's cool ah this is cool um okay what if we what if we i kind of want to leave this because i think that's cool we'll come down here um we'll try a couple more i feel like this tutorial has gotten super long i'm just having a good time i'm just having a good time seeing the results this is cool and this is not really that long of a trained model i could keep going on this one um this is weird it like closes it out and then adds another layer does all this what the heck what are you doing i really wanted is this to work uh makes me sad okay let's try oh wait we didn't um i got rid of the imports i kind of want to leave those cool all right we'll try one more i'm not sure really what else to try um what did you do so we went to layer two uh no dropout model dot add i actually think that should work um give it categorical ah model model name model compile it's close it's just not great and like the autocomplete what did it give us yeah give us like this which is close but not not really um oh and in fact actually i think we passed with the second layer so really all it did was add this not so good um okay i'm trying to think if there's really anything else i wanted to show for the autocomplete but i think that's probably like what if we did this look i'm going to try one more thing like what if we do sequential come back up here we'll do this one this is the one i think i see more often but let's see what happens like oh it just wants to immediately go to like this like create model function which is kind of weird this might be like overly represented in the data set or something i'm not really sure um what if we do this oh let's do this too like will it at least like finish this line successfully hopefully what you got for me is that correct strides two padding same activation value okay so it actually would at least complete this line nicely 32 5 strides two um yeah okay yeah all right i think i'm gonna stop it here um uh like i said the model is uploaded so if you want to try your own autocompletes on all kinds of different things uh feel free to be honest i'm i'm kind of impressed i mean i think this is a somewhat decent auto complete uh i think the biggest problem is even on my my gpu a titan rtx that i'm running it on right now uh it took forever now we don't have to generate like if we are just trying to do uh line complete um where is man we we've been busy today oh my gosh did i miss it um you know there's no reason why we need to do 512. um that's kind of silly so you know maybe change that uh i might tinker a little bit more with this and uh you know who knows some other things too let's try 75 we'll just save that real quick come down here because i think it should be sub second at least what is it too long okay ah it's longer than 75 huh i'm getting sleepy um okay i'm gonna be done now um so anyway there are other things that we can do i kind of want to tinker with the uh docstring idea and do like transfer learning so maybe i'll do that not really sure like i said the model's up there so if you want to try that you want to fine tune the model to your own ideas feel free overall i'm pretty impressed with this to be honest i didn't you know i trained it on the uh the dgx a100 uh over there um or the dgx station a100 i trained it for uh maybe like four ish days uh i didn't even make it through one whole epoch yet so it still definitely could be trained even longer but i just got too antsy to i just really wanted to play with it so um these are pretty good results so far i'm sure i could make it even better i think the next thing i'm going to do is try to transfer learn from the actual trained gpt2 model and transfer it to my data set i kind of like to see how that how that works so anyways more tinkering to be done hope you guys enjoyed this is a definitely a long one questions comments concerns whatever feel free to leave those below otherwise i will see you guys in another video you