George Hotz | Programming | tinygrad: tiny efficientnet is recognize cat? | Part2

Key Takeaways

morning good morning good morning yeah no nudity on stream no nudity on stream don't pull another entity that's going to be the opening on the youtube all right boys i'll show you what we got today we had a beautiful pull request merged this morning on tiny grad made me very happy so uh you know that's why we're why we're streaming uh this morning this from marcel uh bischoff turning these crappy triple loop into this doesn't this line make a lot of sense np.insum i don't really understand how line sums work but good thing marcel bischoff does so i merged in this pull request and then i made a few uh cleanups if i do that will it show it side to side how do i get it so go split oh i didn't know there was a button i guess it picks that sometimes and picks the other one sometimes split is so much easier to read unified is so hard to read okay okay okay okay okay let's see do we have did anyone actually no one's here oh bread ham and egg good morning good morning good morning okay um so we're working on tiny grad today now i've written tiny efficientnet um so let's just go over the code quickly of what tiny efficient net is and thanks to marcel bischoff it now runs in under a second so we can actually debug this now right now so we'll start with where we are on the stream right now here let me uh look at the picture of the cat um sorry it's weird colors but you know so you see this picture of a cat this is the picture that we're putting into the efficient net it's pretty unmistakably a cat uh but there has to be bugs in the efficient net because of course it's saying that it is a coho and i didn't know what a coho was until i googled it but it's a salmon so it thinks that this is that and uh wow look how visual my channel is today wow this is the kind of content that's going to bring in new viewers you know because i'm putting a picture of cat because we're going to recognize cat but it actually thinks cat is coho but it knows that it's not dough so that's good you just got an email that i'm live you can come watch my stream uh yo yo yo no i have a story that's not appropriate for stream i'll tell it later um what i'm busy watching oh you're watching my stream you can comment we got some people from poland yeah the left cat looks weird because it's not a cat it's a fish a coho is a fish it's a type of salmon honestly i'm like curious does someone make like a visual image net explorer oh yeah look wow wow what wow this is this is a lot of um yeah look there's men in caps holding co-hosts all right well either way there's got to be a bug in the efficient net because it thinks that the fish is a cat oh look at these look at these co-hosts wow look at that happy coho it looks happy it's getting away catch it eat it fry it okay so we'll comment out the thing where we look at the cat because we don't need to look at the cat we know the cat is in there it reads the cat from this website here so we can take a look yeah that's what the cat looks like in color and uncropped not a hot dog one is the accord guidera stream i'm not doing that shit anymore we write uh we write tiny grad no i don't think it is a catfish okay there is there is bug yes nice logo the logo was contributed by let's figure out who contributed the logo tsm tsm callister contributed the beautiful the beautiful tiny grad logo tiny grads on pie pie we've made so much progress on tiny grad um now so for those that don't know what an efficient net is we'll start with what an efficient that is efficient rethinking model scaling for convolutional neural networks there's one chart that really explains what an efficient net is and it's right here so these are all the other old shitty networks this is the number of parameters and this is the top one accuracy so you can see the red line is well to the upper left upper left is the direction we want to go because parameters are bad and accuracy is good all right capisce does everyone get it good um how much would you pay i was thinking about doing like a high-end like programming tutoring service i would like like like 500 an hour i'll tutor some rich kid on how to program the problem with that is i realized i'd get fanboys and fanboys would be interested in asking me fucking personal questions so that wouldn't be appropriate and that's why we decided not to do it you want to build a cpu out of relays well i don't know what a relay is if you want to use a relay well you should use a transistor because it's the current year it's the current year um now i'd be happy with 500 an hour and i want to like tutor some like spoil rich kid on programming no no no i don't want a course and the problem is oh my god we get fanboys and fanboys are the absolute worst you know fanboys are like ugh oh the most likely do you buy like action figures and shit too are you a grown man who buys action figures ugh reddit or trash you know can i make the text bigger is that big enough for you well if it's not too bad i went to the eye doctor and it turns out i have an astigmatism and uh yeah so that's that happened all right so we implemented tiny efficient net i'm in tiny efficient that is pretty short code uh so this is it uses these mb com blocks which i think are like mobile bottlenecks and that's the forward pass for it and then efficient that so let's just compare it to the pi torch one and let's just see if we made any uh mistakes so here's that same thing written in pytorch we're loading the weights in from pi torch as well here's the code that does it um so we're just checking inference and maybe if we have time we'll check the backward pass too but our goal on this stream for anyone who's like what's he doing today is to get instead of it saying coho here for it to say cat right so these are the image net these are all the things that could possibly recognize oh that copy paste not going to work and if i do that and then i do that and then i remember to add the tea i forgot to copy now it'll work okay okay so these are all the things that could possibly recognize it at we have egyptian cats we have persian cats we have tiger cats we have siamese cats i don't know what a pole cat is we have madagascar cats and catamarans so those are all the things we could possibly recognize it as these are just the imagenet categories so that would be nice if that worked at all but for some reason it's a coho and what's even sadder is i think it's even a coho if i multiply the image by zero which is of course putting nothing in yeah it's still a coho so you know the image is actually not making any difference at all almost to the point that i'm scared we're not actually putting in the image what if i multiply the image by a thousand does it do something now oh it's a bald eagle wow what if i just multiply it by 256 it's a coho but if you multiply it by a thousand it's a bald eagle and it's really sure of it this is the most patriotic neural network and we'll also put in what it's not it's not a french loaf well i wouldn't want to be a french loaf either all right well we're coming out the knot for now because you know should i watch the new borat is it actually any good okay um also we're gonna listen to some copyright free music all right um because we're not allowed to listen to copyrighted music so we're gonna listen to copyright free music so we don't get muted royalty free music let's listen to this all right some copyright free music oh i listened to the uh alex jones on uh on joe rogan i liked i liked you know alex jones the guy could use a fact checker [Music] like unlike most media i don't actually think alex jones is trying to lie to you a lot of media i think it's they're just evil and they're trying to lie to you i think alex jones just gets caught in his head and talks too fast rogan's kind of saying this but you know i know i know i know alex jones has a point but it's deterred by many substances you know you guys always think i'm on drugs i don't even think he's on drugs i think that's just who he is and like what dude i don't care about your torque mod on the accord i don't care this is my care um it's not my fridge [Music] well copyright free music is kind of mediocre [Music] all right i mean i can run it six more times but it's still gonna say coho so we're gonna have to really start diving in and figuring out you know where the bugs at [Music] and then we're gonna make it take oh you know what we can do this now let's say if all right let's say url equals this uh if sys.org length system v greater than one now all right we're ready to recognize things that aren't cats um [Music] but we're not ready yet because it still thinks the cat's called so yeah also by the way check out tinygrad if you haven't it's actually very readable code um it's the tiniest neural network library here all the ops right here uh i mean it supports everything required for an efficient net well maybe i mean there might be some bugs there clearly are some bugs so uh yeah let's um okay so x squeezed oh yeah i learned what adaptive average pool meant and it actually just means um basically the whole thing is this size is adaptive so it should match that okay we're doing se reduce we're doing swish we're doing se expand and then we're multiplying the sigmoid and then we're doing the point wise convolution we don't oh oh did we not do the skip connection what's id skip block args id skip oh interesting i don't think we did that oh well that would make sense why this didn't work um okay well ah well if we forgot this this would actually explain a lot let's call this inputs okay let's just do it if the shapes are equal x equals x dot add inputs hmm cat cat cat mouse trap oh well we made progress i think i don't know if we're actually supposed to add all these okay let's take a look at what's here what's id skip and when does it get set oh damn okay um i actually checked out that efficient net so we can take a look at it probably in fun there we go um my python's broken so you have to type ipython3 whenever you want to use it are you tails oh termed a relative import with no parent package this is terrible i somehow got this to work before like i don't get it because these block args well we can try to read it block args iv skip and copy and pasted this from here so complicated though um block decoder dot decode see this called so confusing okay if not no skip in block string so we definitely don't have any no skips i think that's actually probably right then checking if self. oh dot stride input filters equals output filters okay that should be the same so i think i actually did that right um input filters output filters strides print it because if we strides and the sizes won't be the same so we really can just check the shapes i believe yeah cool okay well good we added that in see you get one little thing wrong everything's a resnet like architecture today um see what else we did wrong all right let's make sure these things match uh expansion and depth wise convolution so we do the expand cone uh i assume expand comp doesn't have a bias i mean yeah if it had a bias we would and they don't put you don't want to put biases in your comms if they're right before batch norms um because the batch norm already contains the bias and you're just stupidly wasting model parameters you don't want to do that um depth noise comes we do groups that's good we do pad i mean the pads all have to be right otherwise it just wouldn't compile uh we're running bn1 we're running swish we're running this junk we do the pointwise convolution and then we don't need to worry about drop connect because that doesn't actually run though i'm interested oh i guess drop connect will just zero it out okay cool drop connect is kind of like you just get rid of that whole like rez block it's a it's a cute drop out like trick alright let's read forward and see if we made mistakes there god i don't even know what any of that is okay self dot extract features okay well we're not running that um oh maybe maybe that's just the actual net okay we do an average pooling uh we do a flatten which is this reshape here dropout does nothing and then we do the fc oh why am i doing a swish there don't do a swish on the output why do i think doing a swish on the output was a good idea anyway no swish let's see still mousetrap [Music] it's not dough good let's see if we zero out the image if we get something different this time okay okay we're making progress a zeroed out image is a suit that's right if you're empty and hollow inside you're a suit what's uh who's that guy billy from uh entourage a soup all right um there's some chance that i like mess up indexing these things too so like we can't always rely on just uh if like next to mousetrap was cat yeah because they start with one they don't start with zero oh yeah i think this is off by one yeah i think it's off by one i'm actually not even sure if this is the order there's some chance that just the is wrong uh let's take a look at a plot who likes plots you like plots i like when things have plots too plt is not defined i'll just put that there is that slow what does that seem like something that would be slow importing matplotlib now let's just move it down here if you want to look at the outputs well that's junk um it's probably because it's a bad shape yeah it's a dumb shape all right let's do out.data sub zero let's see probably unlikely to be an indexing issue i mean so this is like these are like the log odds of each uh of each thing now there are a whole bunch of cats in there so i'm not sure let's look at the log odds of a zero dot image those actually look very similar it's a suit a mousetrap well at least some of the image is making it through the net whoops all right let's go back to reading look we found two bugs already i was doing a swish wait well that swish is good forward just that other swish was bad dot fc add fc that's perfect let's print fc bias all right relatively small numbers that's good it's like the base rate of the category you know um okay so we do an average pooling we do the reshape first no we do an average pool there and then we do the reshape that's you know what i know it causes a line but let's do that good all right 1280 is the output we do reshape let's read extract features and see if we're running it right okay um we have the stem bm stem just try to i guess that's right uh drop connect run the block oh let's just say block you know the block is hot i'd love to play the block as hot but boys it's copyrighted so you know can't play the block as hot all right extract features doing swish bn1 conv 2d on the conv head okay that seems right extract end points when is this used does this even do oh i say useless let's commit this since we know it's bug fixes fix e-net bugs now is mousetrap so i don't actually know if that categories thing is right maybe there's an example in here where they actually fetch the categories um like by name this is useless examples maybe imagenet preliminary data how do i get it to actually print the name of the category whoa this crazy amount of stuff okay this looks the same zoom from a checkpoint distributed oh look at that those look nice what's an average meter i should read more pi torch code but how do i get the name of the category you guys saying this in chat uh yeah i know git commit am will do the ad for me but then i don't have as much control uh i used to do that but then you accidentally end up committing dumb crap and that's why i don't do that yeah we're using a pre-trained net and we're trying to run it in tiny grad deep research that's what we're doing you're a subscriber so you get your questions answered by the way if you haven't had a chance to subscribe yet make sure to subscribe it is only because i'm a whore for money that i stream on twitch somebody has to pay to keep up my math habit and you know without your money i would stop doing math and that would be a bad thing for the world so um oh here's a clip george hot does meth we'll put it in the new york post oh is that uh was that the uh george hot ai expert news let's see if we can find it no no no no no there was a news about me where'd the news go said something like aliens too busy having sex duh there we go the daily star yes yes aliens they could yeah aliens are too busy pleasuring themselves to visit earth an ai expert believes someone asked me if i like being called an ai expert and what i said is i want to be called something like you know a little scary and definitely with no prestige like hacker is much better an ai expert seems like the kind of person who's like well i'm going to go to a conference and i'm going to lecture people about my expert opinions i'm a thought leader and that's what they are like so you know i never want to be grouped in with those people that's right even though i am writing the hottest growing neural network library called tiny grad um you can use tiny grad tiny grad is great you know what's great about tiny grad is totally and completely useless there's no reason you should actually use tiny grad for anything but we still write it i'm getting such good quality pull requests for tiny grad oh it's an art project boys you know all right um quality articles as usual am i at home or the office i'm in the cloud all right well it's pretty confident it's a mouse trap and i don't think that's right oh don't make me don't make me actually start to have to like compare it layer by layer to pie torch i have better things to do today you know i do believe i have better things to do today we're filling in everything so that can't be wrong i tested my batch norm so that can't be wrong what else could be wrong where are the bugs i didn't of course so i have bad posture and it's because i have an astigmatism and i'm going to get glasses and you guys can see me with glasses okay so clearly we did something else wrong because we're doing a swish at the end here and there is no swish here is there not a swish here did i do the same thing do i just love putting swishers everywhere it doesn't look like there's a swish there so i have a swish there and let's get rid of it uh everything every bug i fix in the block is way more likely to fix things oh let's go let's go cat cat cat pick up ooh but it's very confident about pickup so i'm interested if now we're in off by one land because yeah we have to be we have to be off by one let's see if pickup is next to cat oh yeah oh if it says cat right now if that was the only bug i'm gonna be so happy because then we don't have to do shitty things like oh well the the confidence is very different though bar ballet well that's good how different that is so it could now just be let's look at the outputs and let's see if it's very confident about cat oh it's very confident about that one okay all right i think we've got it i think we just need to figure out how the categories are are mixed up i'm hoping notice how it's yeah yeah okay let's um let's make it work on some other images too let's find some other images like a car california association or realtors eh this is a copyrighted image do we need to blur that image out on stream what oh my god stupid google what okay well that one i'm getting other things for oh that's a bow tie great let's take a look at the car and see how much we're messing it up oh no that's pretty good actually let me write this to not be stupid with this uh 87 thing print image dot size let's just write my image resizer to be better one sec uh okay so we'll say aspect ratio equals image dot size sub zero divided by image dot sine sub 1 we'll say this is inch 224 times aspect ratio do it right boys do it right don't write bugs okay so now we have that now we want to put here um oh god we want to say uh i'll call it the chapo image dot shape sub 1 minus 224 oh what if that doesn't round nicely oh you know what better idea called call it image.shape.1 minus 224. you have hardcoded 224 everywhere that's a tragedy uh divided by divided by 2 for that integer division and we'll just say chapo plus 224. all right let's take a look at the cat and in this case it's a car oh beautiful car uh what do we think is that a beautiful car all right good cleaned up the resize logic okay it's nice again because it's very confident about this and it's very confident that it's something else which means we probably just have the labels mixed up because it's not really a bow tie is it let's check out bow tie and if there's car near there no they're probably just all wrong efficient net category list i mean this code has to be in tons of like stupid like oh i can recognize a dog efficient net uh category list you see what i mean about how it's very confident in the bow tie um all right category list does anyone find what i'm looking for no you guys are useless gucci flip flops google is not stupid i beg to differ i mean what do you mean by google um i just found that list somewhere on the internet so that might very well be wrong uh let's not say efficient now let's say imagenet category list here we go does this match this one totally different uh oh do i have laundry i don't want to miss out on this um i don't think so though oh yeah but i put it back in the thing i put in my my spot let's see maybe these labels work better i do like that they start with zero that's a big improvement mwah uh is this json oh what am i doing i'm doing a dumps it's dumps i'm doing loads i'm not doing a dumps expecting property name enclosed in double quotes oh i see it doesn't like that do i really just want to do an eval we could just do an eval i don't know you know what someone probably already wrote this code before let's just google it ah here we go literal eval huh what's a literal eval safely exhalate an expression oh it says it's safe let's trust it let's blindly trust it uh what shoved in the bag yeah yeah yeah if it's like shoved in if it doesn't look folded it's bad oh ast.little eval all right we've got to import asked let's see let's see what we got oh beautiful that looks good no wait that's junk what why did i put all these dots in there junk just added a lot of slashes oh value error malformed node or string it doesn't look very malformed to me that's malformed about it decode utf-8 oh what you don't like my exit what how come exit doesn't work did we get hacked i feel like we got hacked by running ast literally val who redefined exit i don't know i feel like that used to work but what do i know yo boys look what it says that is it said that's a sports car can't make that shit up you know that's a one in a thousand chance if this one's a cat this means it works one in a million let's go tabby cat we did it oh that was so easy oh it's a short stream yes please oh we did it oh that was so easy oh yeah sports cat yeah all right someone want to post some other pictures nothing inappropriate but let's post some other pictures and we can try that resnet comment is stupid i feel dumb for writing it we're going to delete it oh man i can't believe it was that easy we just had to delete a few swishes yeah e-net works all right um that's a broken is that a broken link oh okay we got some air jordans up in here all right let's try it let's see what it says it is by the way this code's written 100 from scratch just numpy and crap running shoe oh yo yo ai man who's scared of the ai that's right it's coming for you it knows about cats cars and shoes i've never really played with pre-trained imagenet models before it's impressive and it's impressive that you can do it with just like you know like nothing basically oh and it's fast thanks to marcel bischoff oh we got a rooster hawaii let's see what we got no that's a fi oh oh we need the image all right let's go rooster it's a cock oh yeah yo oh man look it works so well and it's written all in tiny grad did you believe that with just you know you know what you know what's crazy about this this thing works for training too if we could just send tiny grad ten years back in time and someone had a big computer to run it on like it is amazing that that's all it is can it do hot dogs post a hot dog picture what do we got i don't know what it's going to say for that oh i think it's actually going to error out because that image is taller than it is wide yeah all right all right we'll fix the bug let's fix the bug um we want to say aspect ratio that's a lot of bug defects sorry it only works with images that are wider than they are long if someone wants to submit a pull request they can all right hot dog let's see let's see if it knows hot dog i want i can't do that hot dog hot dog red hot this is this is and it runs in under a second it's amazing that this works this is like all the code and you need to know what these things are like i guess you have to know what a batch norm is um and this should be able to train as well all right we can get rid of that and get rid of that load weights from a rough copy of that yeah i like wait what i'll get better there let's see what code we can clean up does it actually need the squeeze what if i get rid of the squeeze does it still work nope it doesn't work now it thinks it's a three-toed sloth make sure to keep your squeeze in there so this actually probably doesn't work on the backwards pass yet because i didn't write if you're doing uh dementia if you're doing broadcasting the forward op support broadcasting and the backwood ops don't so that's a big project to fix but yeah you don't need this this is literally and you don't need any of this image pre-processing it's all fake um it's just amazing how far things have come that you can write like a toy under a hundred line project uh you just need to if you rewrote those things fast on a gpu and you made that work um you could you could train this net you know 20 years ago maybe 20 years ago 10 years ago definitely 10 years ago definitely if we could just send this back in time 10 years we could usher in the deep learning revolution you know if we could just send tiny grad back in time how fast does it run on a cpu well so it runs in point seven seconds let's uh let's just see if it with bigger things uh let's say like 128 access zero like what if i do batch size 128 oh my god it takes forever by the way a batch size 128 efficient at b0 on a gpu takes like uh who who sent in appropriate links we should ban them send it back in time the ai will rise sooner than because if we send it back to 1969 the people would just be frustrated they'd be like what the hell is python ah how many other deep learning network how many other deep learning frameworks have efficient net fully implemented you need a lot of gpu memory not that much i don't know you know yo you know what i just ordered 8 30 90s i got eight 30 90s coming no it's not using the gpu this is just using the cpu and it's incredibly slow this is running on mac do you think quantum computing will kill bitcoin sometimes just be banned um subscribe and i'll answer your question how much ram on the mac 16 i think thinking can i cap proc no of course not can you do about this mac and you guys like looking um yeah 16. oh no i leaked my serial number you guys are going to claim my warranty uh you know what honestly i stopped doing the oculus hacking because i hate it so much um i thought they weren't in stock anyway yeah that's why you pay scalpers absurd fees and you hate them the whole time what do i think of the new the new one the new atis wow okay that is terrible it's actually slower to do that than it is to do that it's actually like it's quicker if you're trying to do a batch to just do them all yeah try it on your linux vm with four gigs of ram it's gonna work great what do i think about risk five uh is like civilization five um is self-play halfway to learning dynamics oh i'm learning a dynamics model right now at comma we're learning dynamics models now on the latest vision features we just upgraded everything to ubuntu 2004 pi torch 1.7 the world is getting so good the world is getting so good how do we automate away middle management we just fire them i don't have any middle management fired them all all right so i think tiny grad um all right let's let's go over the to do's to do uh efficient net backwards pass uh what you do train an efficient net how insane would it be if we could actually train an efficient net using tiny grad tensors on gpu gpu support must must support mac reduce code increase speed okay uh we can we'll play around with this a little bit let's let's do that a little more for the stream let's look into what it's actually going to take to move tiny grad to the gpu so we can get real performance out of this library you know um so first the only thing that we're going to have to change let's let's do a little refactoring before we think about gpu let's see if we can implement any of these ops in terms of other ops let's move dot down here gem sorry it's hard to refactor with the text big for you people how i do a mean i do a mean somewhere how do i do a mean if i can't divide let's look in the tensor library and see how i actually do that because divide doesn't work oh maul div yeah okay so we do the reciprocal here i get it and the reason that we have to do because the devop is kind of flaky um i'm not exactly sure why but i don't know something weird about that gradient so we probably actually want to implement div but then we'd have to figure out what the derivative of div is and i don't know how to do that okay uh add some mole pal and sub maul actually should we make sub just be a maul with we could just make sub a mall and an ad no then we're actually doing a multiply okay let's make it an invid ad uh what is that op called where it's just a unary minus sign uh uterine negative i mean okay theoretically we can implement that just in terms of those things but i don't know no never mind is boring those are fine i'm just trying to figure out like what the minimal set of ops is all right so we definitely need an ad we don't technically need a sub but i don't actually want to implement it in terms of maul we definitely need a pal like a sum is a totally different thing so that's fine um we need a dot we need a pad we need a relu um all right let's call these like uh i'm gonna call them like simple ops put pad 2d and reshape and then activation ops relu sigmoid large softmax we have the calm and we have a couple pool amps cool we're just going to call them activation ops so convox one two three one two three four five six seven six seven eight nine ten eleven twelve thirteen fourteen okay so we have to implement 14 ops now unfortunately so one beautiful thing about uh about things like the qualcomm snapdragon is that the gpu and cpu have unified memory so i think that's true for the intel graph it's probably not even true for the intel graphics it seems like it has its own memory i don't know but either way we have an amd radeon pro in here and that has four gigabytes of uh of uh normal memory so that's sad we do have to make like broadcasting semantics work i will add that to the to do make broadcasting work on the backward pass simple please so you guys know what broadcasting is if you don't know what broadcasting is get ready i'm about to educate you okay uh so let's make a whole bunch of zeros let's make like ten zeros right uh important on pi as np all right cool we got ten zeros right notice how i can add a one to ten zeros i just broadcasted that one across the 10 right i can do it even if i make np dot ones one right and then here i can add three to this and beautiful look it's four on all of them right um and i can even broadcast you know i can broadcast on uh on axes right so if i do that and then i say like np dot array this isn't going to work exactly but i'm adding two to the first one and one to the first row that's not going to work because i have to reshape this but if i reshape it like this we're going to get a row of twos in a row of ones like that right so that's called broadcasting now unfortunately my broadcasting doesn't work on backward pass so we have to make that work um what has to be broadcastable yeah probably everything i'm not like the pools and stuff probably just the basic ops because i'm not a real twitch streamer well i don't have a sub counter oh that double line is so annoying mr pool can be so annoying who remembers what that's from that is so true i can write that better but whatever see what other lines are absurdly long this line here is absolutely long the line's long too but i don't really think there's a better way to write that line yeah that's uglier remember guys tiny grad is a work of art so the code has to be beautiful so yeah we're gonna have to rewrite this on the gpu if we want to make training actually work let's run the ops test to make sure it didn't break anything looks good unfortunately my devop is so i'm gonna bad out again more white space all right hi opencl gpu tensor library i mean you know we really could we could back the whole thing by uh we could back the whole thing by um you know what you would call it so we could bag it by uh pie torch but that would be cheating um yeah so okay pi open cl oh wrong pip so i think everything would work if we just used well not exactly we'd have to make a small change to tensor then we have to dispatch the ops depending on whether we want to put them on the gpu or not on the gpu so we'd have to make gpu tensors work okay we now have pi opencl let's play around salvdelg thank you for subscribing import pi opencl as cl this looks fun if we can make it work totally without numpy and just cl no because we still want to pass any things in numpy we want we want our creation ops to be a numpy we don't have to go crazy with uh creation ops will require numpy and then we can write code like this which is actually kind of really nice um wow okay that works really well with uh okay i think we should do this work in a branch start with a branch called gpu and then we want to add a dot cuda method yeah take that nvidia dot kudo what do you think um don't try uh except around this even though it's not cuda that's right we're gonna genericize cuda just like kleenex yeah let's genericize cuda yeah accept import error pass no gpu support all right so we have dot cuda um and if we dot coo to something we're going to want to copy the buffer to the gpu so oh we need a context um where should we create the cl context maybe up here is it really called create some context i think i have options what oh my god this is so hard i don't like create some context hmm no when interactive is true i can just say answers has no attribute pop oh cool can i do minus one no minus one okay um change if you don't have mac okay we've created a context now we need to make a buffer let's when you do dot cuda dot cuda i wonder what cuda is it means generic gpu computation yeah um self.data equals cl dot buffer host buff equals self.data yeah okay so now should we try out our nice tiny grad um from tiny grad dot tensor import tensor oh tensor we should do this in a notebook warning it's not float32 really what am i doing i can just say tensor.0 is 45 that's better code yeah use your abstractions all right cool mf is not defined cl.memflags oh great okay let's do this in a notebook uh did i create do i set the path right yeah yeah good okay good so now we can develop this in a notebook uh tensor dot zeros all right we have a z tensor now we can say z dot cuda oh let's see what happened oh pi opencl buffer what if i call it again ah bites like object okay uh we should prevent that uh self dot gpu is false if not self.gpu and then we also need a cpu so this actually moves it in torch it makes a copy how do i feel about that we'll see we'll understand about that later [Music] i think cuda return self um then we have to figure out how to copy it back so how do we copy it back great very complicated and we have to create a cue as well we'll create a global cue okay so we have to copy this back from the gpu i don't know if i like how i'm doing this but let's play with a little bit more and see who we're doing okay that creates a buffer where does a.n.p come from oh it's here you can't just do that can i can i what if i do z dot data expected an indented block let's move to crop cookies who has cookies steal my token right but now where'd my chat go really there's my chat other windows not just that one great how many people are watching right now i can't say um 641 cool okay i think we're gonna have to track the shape out of band the size is into zero what well one hundred um assert ah self.data.dtype equals np.float32 um only float32 on gpu uh and then we're gonna also say yeah i don't really like this you shouldn't be able to change the shape so we're going to say we're actually just going to change the api a little bit here we're going to say self.shape equals data.shape and we can remove that all right and then shape will still work so we're gonna need a when we copy it back um we're going to say self.data equals np dot empty self.shape uh d type equals np.o32 and then we'll just say cl.nq copy in the queue we want to copy from oh no we just close data data self.data cell.data equals data return itself okay so now we have cuda and cpu just like torch so let's just test sakuda cpu uh pi opencl buffer has no what cool works well oh we have to set self.gpu chad i don't understand how that worked at all oh it didn't work good all right we have copying tensors to the gpu and back from the gpu working now we're gonna have to modify the dispatcher to dispatch for gpu ops oh this is so easy copy tensors to and from gpo sweet why does the efficient net example crop instead of resize and stretch because stretching is disgusting you know sometimes it makes people fat sometimes it makes people skinny either way no one's happy if you get cropped you can't really complain yeah i hope you guys are trying out tiny grad you know tiny grad is a great open source project and it now runs efficient nets so you guys should try it we're going to add gpu support yeah yeah i'm talking to them how's curtains going did you get the right job or you're gonna wobble the small drill bit around to make a big hole that's right always get the right tools you'll feel better you'll make the perfect hole you'll feel so satisfied okay your one yeah yeah we gotta drive to la boys you know going to halloween party tonight in the presidential suite is riffraff too copyrighted to play uh no riffraff's not copyrighted there we go look man if you want to dmca me and take me down do it do it you know what you can live life afraid of the government or you cannot and i don't live life afraid of the government but you know i'll probably end up in jail for it one day but hey man you know uh what by not being afraid of the government people are scared of the government man i'm not scared of government yes that's right they don't like they don't like new york i know i know i know but you know it's a free country at least that's what the government told me free martin shkreli i agree with that i agree you know we were talking about who we should pardon he's on the top of the list donald trump if you're listening i will actually go vote for you if you pardon martin shkreli alex is sorry but she needs to get this done tensor has no attribute ad why does sometime they not get registered um hmm whatever we'll hack this for now unsupported what what what are you saying alex i'm busy now look guys if i get banned it's you who lose out not me you know i was kidding about buying meth by the way i've never bought meth in my life certainly not with twitch subscriber dollars all right so we have to write an add operation let's write an add operation um we got a pull request what'd we get a little stretch instead of crop fix temp location for windows oh that's disgusting wait can't we just get tempter doesn't this work on linux too is temp file built into python or is this some library that i now have to have yeah yeah um yeah you want to change that uh you just don't don't i'm sorry i'm not logged into github right now but just don't put the uh modifier to not be different on linux um you don't need that if statement the temp file one is better and more pythonic uh no no no that's no i'm not doing that what the guy's trying to stretch my images you know someone's going to be you know someone's going to look fat in a stretched image they're going to sue you know we just don't want to take that kind of risk man what oh you hit concrete you're a masonry drill bit alex i don't know what to tell you this is your project what i never complained about anything masonry drill bits usually have a little thingy on them yeah yeah hmm so ctx is a function let's just add it here let's say clctx equals ctx does that work um we can also say cl q equals q you know what let's not just call these generically let's get ad working on the gpu and then i think i'm happy and that's the stream because i gotta go to la we got a party to go to what what do you mean we're very cool very cool man if i was in high school now shit i was popular the first time in high school i've been watching sabrina the teenage witch right so this is what i've been thinking about a lot being cool in high school and stuff um yo high school was was the best time of my life that's why i was into emo music you guys ever see the show do over where joel pretends that he wrote time of your life by green day because he went back in time that's a show joel he was in the show called do over let's see if we can find it um about a man who gets to relive his childhood joel yeah i knew his name the series begins showing an adult joel larson as a single depressed paper salesman disappointed with how his life turned out well no no no i signed up for hulu i know i you know who signs up for hulu but um i am not a bug man yo all right all right harassment architecture speak speaking about bug man um all right i forgot we gotta write actual code here let's go let's just copy the opencl example rig we have to create a write-only buffer should not be a dot n p it should be this is x not size is that right rig dots um we should probably pre-compile this to do pre-compile on import question mark yeah maybe not all right we want to prix some it's not the ctx we have to say cl say ctx dot cl ctx ctx.clcgx ctx.clq oh i can't do a dot shape there but again i think if i just do x out size that's fine none uh x y res g return res g let's try that bye bye chicken picture why are you guys highlighting messages this looks like it's time for sub only chat inconsistent use of tabs and spaces why don't you just order an apartment with curtains luma came with curtains yeah but it came with curtains and douchebags actually no most of the people in lumo are pretty nice i don't know about that why would they make why would they if you're just going to make that mf why wouldn't they call it cl.mf you know unable to compute length of object what oh i think do we have to put x dot size in these thingies [Music] ah that is okay [Music] lcl buffer well hm would like duct tape oh you just want if you're screwing into masonry you're gonna break your screws goddammit george has 41 000 followers of course he was cool in high school that's right that's right you know because like bowling for soup said high school never ends and i'd let you listen to that but it's copyrighted i'd also let you listen to the candyman you know no we're gonna listen to the candy man you know the candyman can you know that song gonna we're gonna yeah you guys can hear me i hope you enjoyed wonka time is that the new vegas song wonka time um oh the tunnel scene oh yeah um why is this an invalid value i don't get it hi opencl buffer we can check the size it's 40. um we're printing stuff okay created that now why can't i go to cpu clnq read buffer failed invalid value what's invalid that's how it does in the example right we return res g right only we don't copy any buffer and some submit it to the queue claim you copy empty like why doesn't that work oh this is n bytes i see my size times four because it's float32 um yeah so this is still wrong shape is not data.size shape is dated at size over four well it's not a registered thing i want to add there we go all right aside from the fact that it's way too big for a reason i don't really understand because that should just be made with shape right so what if i say z three dot shape so not temp somehow 40. that's 40. that four does not get loaded but i divided it by four or am i not needing to multiply that in somewhere i'm actually making the buffer way too big uh no they do n bytes right x dot size oh no x dot size okay i don't need to change that because that's the gpu tensor okay look at this we have an ad on a gpu whoa that's wild look at that tiny grad can add on gpu add on gpu um all right should we uh should we try some speed benchmarks so let's just let's add a whole bunch of times what invalid arg size a oh that's not right we actually don't want to enqueue the size because that's going to run off into junk memory we want to enqueue just four of them divide by four you see why we have to divide by four because it's like mixing up bytes and not bytes okay still wrong 10 cz2 data that size okay why does that have size to oh no oh because we converted to the cpu oh that's stupid um it should really make a copy let's make a copy instead of doing that should make a copy right sometimes i'll just set that we just return tensor data we do want to make a copy and return the copy ah two three four all right let's make big arrays and now let's do time it not time out probably triggered reimport so let's do that unbound local variable oh stupid adding gpu so why no rock m it's not actually cuda it's opencl oh my god we're actually dispatching a kernel for each one of those ads that's terrible this is going to be so slow why can't time it just do the smart thing star child is that a real name what separates setup code from code in the first line of setup code oh my god it's printing so many things it's disgusting oh there's no tear down code okay so that's how fast add is on the cpu now let's check it on the gpo what oh it's not gonna work because we don't have cpu ops enabled terrible terrible who wrote this terrible code well i don't know that seems pretty fast to me it's doing it in it's adding numbers so fast adding works yeah i'm doing low text yeah and it's it's bad uh opencl is cool we are ahead of the curve i really hope this supports opencl 2.1 so we can use all the latest opencl features and apple i know you want to deprecate opencl and i'm telling you right now don't do it usually you make good moves look at how you've moved the world to usbc look at how you've moved the world away from cd-roms look at how you've moved the world to so many great standards yet you insist on shitty metal consider ditching metal and moving back to opencl this is my plea to you you can make opencl the standard for the future and not fragment the ecosystem microsoft consider moving away from directx i know this is going to be a much harder sell for you but the whole world supports opengl now you know what's beautiful about opengl and opencl the openness you know what's terrible about directx and metal the closeness the fragmentation the wasted hours of millions of hours wasted on your fragmentation because you insist in having your own stupid standard it's not better it's worse for humanity ditch metal go back to opencl there's no tear down code too big look at that it turns out if you add zero if you add one to zero ten times you get tens okay how much do we need to write in order to support tiny bobnet well we need to write a gem we definitely need to write that oh my god all the all the tests are gonna fail wow all right well let's let's work more on the you guys can write the ops i believe in your ability to write the ops um for the gpu branch but let me work more on the actual skeleton of things and then you guys can fill in all the ops with pull requests by doing pull requests to the gpu branches we can get the tests all working one at a time again oh this is going to be so enjoyable for everybody involved yeah i'll work on the skeleton of things maybe a little bit i gotta fix that shake issue [Music] that would be kind of nice and then that would be a better uh maybe a cleaner way to support this so right now the problem is that you can't do z dot data dot shape because pi opencl cl buffer has no attribute shape we might be able to do this is that going to work we can say z.day to that shape oh hell yeah let's go let's go all right so now we can get rid of we'll add shape back as a property make sure every whenever we create a gpu tensor we set the shape whenever we actually create a buffer we have to set the shape so we don't want that anymore that's junk um we don't want that anymore just do else here no that's not true because of that uh we shouldn't check the d type anywhere so that should be okay we just have to make self.data.shapework so where do i create buffers um i create one buffer here so we have to say reggie equals x dot shape uh helper buffer like so we'll make a buffer like thing and then where else do i create buffers and i convert it here so we have to say data.shape equals or actually i could just say self.shape yes yes that should work and now we should be able to add weirdly shaped things well let's make sure it actually does work oh what's 22 oh you know i'm a no guy you know i'm a no you guys don't know that shit no they can't no alex all that is all that is is basically saying give me a thousand dollars give me a thousand dollars i'm not gonna giv