George Hotz | Programming | tinygrad -- Llama 2 70B on a tinybox! | tinygrad.org | refactor needed

Key Takeaways

oh you guys couldn't hear anything I was saying anyway well I said some good stuff Google have to read my lips um why is that GPU special okay are we live or what trash things okay we're gonna pop out the new chat all right good good good Mike muted all right all right all right we're good we're good we're good uh pop out chats make that big all right all the new freestyle oh yo no no we got we got better bars man we got where'd that one go it got taken down for copyright um I'm sorry if you missed touching grass freestyle uh we play touching grass freestyle on here I don't think we can you know I had a bad time with YouTube and copyright lately uh okay so we have some band terms and it's anything to do with conductors that conduct more than the normal amount well all conductors in here must conduct a normal amount of electricity not an abnormal amount um so yeah the new freestyle is gone but we haven't even newer freestyle oh man I think you guys are gonna like it uh someone I don't know reached out to me and said yo I made this beat for you I want a feature from Tom Cruise so I sat there for an hour this morning and I put something together uh I'm glad you like touching grass uh but touching grass is gone forever uh semiconductors are okay you just can't conduct too much man if you're conducting too much you're not welcome in this chat but the problem with this cover is it comes off sometimes and I bought a new chair because I'm not poor but it's gonna take six weeks to come you know some of us don't have six weeks so yeah I got this cover in the meantime and it kind of fits on the chair okay uh okay let's see what actually works on this computer if anything does that might be too much light get this set up we don't need this light do we need this light get that light like common 3x ain't no one at eight weeks don't worry it won't be eight weeks order it right now it's one to eight weeks if they told me one to eight weeks on the chair I would have been happy because at least I could have had hope you know what I mean you just got to give the people hope someone's out there buying lottery tickets your ears oh did I hit something [Music] the microphone's so much more confusing to use they didn't make it idiot proof okay let's see all right so I downloaded uh all right so I downloaded and we have the the official tiny grad meeting in the tiny grab Discord at five o'clock so you guys will be there for it um is that a subscriber symbol no you cheered 100. you you turn that into seven figures with crypto and then lost it all in Luna yeah that sounds right all right subscribe only chat mod PS5 that that's really that's really the I don't give a shit about PlayStations you know what I mean like I'm a 33 year old man like when I was 21 I cared about video games when I'm 33 I don't okay like I'll play some games a little bit but like when you're 21 you got nothing better to do you play video games all day right you play video games you hack video game consoles I will never hack another video game console I just don't care you know what I mean like the 21 year olds can do it it's cool it's cool when you're 21. it's not cool when you're 33 you know what I mean um all right uh don't use guacam you'll regret it you'll just you'll just get ready for a upsetting two years if you choose to use Qualcomm in your product so we have a llama here we gotta install Sam's piece does this know how to load okay so we don't have that llama this should work with the other llamas though right not found weights llama to tokenizer model um I downloaded it over here best token as a model okay we have to create a directory called oh told me here it's actually called Lama two um wait also that's not even true we didn't even have an angel round at that valuation wait what are you even talking about bro this is not even real news oh my God I have to move these this is a lot of work no I I also spent why is it trying to use llvm oh I know why it's trying to use llvm okay okay let's hope this works uh llvm is the only back end that supports B float 16. someone put that in the loader someone probably put that in the loader there's this file called state which is the loader yeah here look at this should this be done in the example yes it probably should um so the back end doesn't support B float 16 so we have to convert it to uh float 16. just take forever to load I haven't tried this yet I don't even know if llama2 works why is this so slow the llvm workaround might be broken on Linux oh there we go this is mad slow all right we gotta get bitcast working I think this stream's actually going to be getting bitcast working and not getting 70b working that's brutally slow okay user hi Stacy okay it works and then it's like slow to run too I don't know why this is so slow no okay so part of the problem is that lovm's linking to uh some float 16 stuff doesn't work probably need to compiler RTM is this tested in how's this how does this work on the CI box let's see what's failing if you guys know about debug and Tiny grad you can just say debug equals four and now it'll yeah okay so this is crashing and it must be this FP trunk thing I think I can do debug equals five and we can see the assembly too yeah okay so we're call queuing what's ever in our yeah so it's not linking to this correctly I don't know I think we just gotta fix big cast [Music] okay so what bitcast is is basically in uh so what was wrong with this PR atch test s okay so what bitcast is is normally when you do a cast you also convert the type so when you do like a float to an end cast do you want that cast to uh change the do you want it like just four and float mean for an INT or do you want the foreign float to literally mean for an ant uh what are you saying I'm on stream that seems like a reasonable test so I'll show you my issue in tiny grad where I talk about this here in B float 16 support this is really what I want it to look like oh but that's like a little Annoying that's not actually going to be that much better because yeah a bit casted to float32 and then I guess I could normal cast it to half and I could do that all on the uh on the GPU okay let's make this work on the GPU it seems fine how good is this pull request I mean can we just start with it as a thing or should I copy and paste that things in one at a time I don't really understand that foreign remember well this should actually be identical with normal cast I think two well these are gonna fail because there is no bitcast but like these ones should actually just be I think the same well okay that's a different one this one is just normal cast so we shouldn't even be supported I don't think yeah those just work okay so these ones are just normal cast we're going to get rid of these as tests but that stuff should work uh different debug levels give specific outputs yeah so debug equals four will print the code that goes into tiny grad debug equals five and six will try to disassemble it and show you the thing after the compiler ran um okay so we don't need the UI you don't actually need a bitcast operator you only really need bitcast operators to go from floats to ants uh so we will check that let's take a look at ml Ops okay we have cast it's a little Annoying that this is just copy paste bullshit but it's probably right yeah I think that's right okay we need to put it in yeah there's not a better way to do that really we need to put it in tensor.pi Yes actually those are the tests don't really hurt I guess we could just leave them because they are actually technically correct there's don't know if we're gonna support them you know what I mean like because the casts are identical okay we have a bit gas function by the way thank you to bedex for this pull request so I can copy it oh okay so which run time do we want this to work on I think we're testing on the GPU so we want to work on the GPU well it's a little Annoying that it's enumerating all like what's in Ops GPU Now isn't really right you don't want this to like pre-fetch like you don't want this to create a context in a queue on every GPU I really only wanted to create it on one I don't even see a way to like like I can exclude it but I don't even see a way to like tell it I only want one this Line's hard to read I see what it is okay so if device is being passed in there you only want to do one I'll just put this here for now so it's faster because otherwise it's creating yeah but there's not an easy way to do that okay lazy buffer object has no bitcast let's go into lazy buffer and write bitcast um it's pretty boring I'm going to add this to unaryops well there's so many places I have to change this can you guys becoming terribly complex and awful um okay now we don't have it so we actually have to write it in GPU so we have to go over to this code Jan well no one can ever this code is becoming unreadable look at how many places I've had to edit it this is as many as clang at least it's short short which of these can be ripped out we're going to update Ops as well there's a lot of lines okay so I don't think this is actually written correctly I think the way that you actually want to write bitcast for these things is you want to create a raw buffer it doesn't even have to make a copy you know what I'm saying like it's just a view okay so the way you really want to do this on the GPU is you want to take a CL buffer and just change D type like that's really where bid cash should exist there's no way to do this with the buffer all right so we're just gonna have to like make a copy but what is this no you do water because you want it to be able to fuse right so you can't actually make a copy oh now I need a bitcast operator in the linearizer all this is becoming insane collapsing under complexity collapsing under complexity yeah just one more thing over and over and over again I just gotta sit down like I mean maybe that's what's frustrating me so much that this all just needs to be refactored cast is not an ALU op casting doesn't even work this cast stuff's just wrong if I add it here I think nothing will get passed through okay it's complaining and saying Kiara and exec AST interpreter doesn't have it but I added that what back end is this on oh I broke the GPO I say so it wasn't using GPU I was trying to use numpy because I broke it by doing that I don't really know why uh okay this is a better error um for the training PR is what I want to see is a trained model that has normal cast work where's cast foreign like I think cast needs to be fixed first right like this isn't ready I shouldn't be doing this now like I'm just hacking stuff right and here's another question like maybe bitcast should just be a flag on cast I think it should be well I don't think it should be its own op and stuff this is a lot of this is a lot of added complexity right and this becomes obvious when you start doing this all right so let's make some changes right and this is this is the tiny grad way you actually reinterpreted was true I'm just going to call the flag bitcast equals false right that's kind of what I'm thinking like you start adding all this stuff and you're like okay wait a second we've done we've done way too much complexity here okay so that's not right um with their tab with their is their white space or something I hate that okay let's get rid of bitcast let's go back to tensor.pi I think what we want to do here is say actually I don't want that to be a flag here I want this to be a flag here let's say apply bitcast equals true now we can go back to ml Ops and we can get rid of bitcast here and we'll have a slot called bitcast and we'll have good cast here okay cool it's a lot simpler I'm not going to make that change to state right now um we also don't need to make any changes to Ops if we're doing it like this okay there's just changes to that which is fine I don't understand what these changes are to linear eyes are are they white space we don't need to use linearizer either we might but we don't even know okay lazy power cast takes two positional arguments but three were given okay so we're gonna have to add it to cast in lazy dot high so that can go I'm going to attach that wait I added as lazy.com um okay so we have an r and the ARG right now is the d-type to a unit reops cast uh should we make it a tuple you might want to make it a tuple you know cool all right now that's fine that's the argument that's going into lazy's cast here so the arm is topple D type pool our element-wise up anyway Ops cast okay that's good now we're going to need to edit these element wise Ops and midcast actually the right thing was surprisingly in the last hour they haven't gotten any better uh and you might want to consider using rock chip let's try it honestly video topple has no attribute NP um true CPU I must just set self dot d type Raw oh it was up unreal Ops cast it's not actually that that was probably more of this crap oh if you bought a tiny box you get an emo that sounds nice okay so what sets self dot d type oh if self.dype is not equal to R Sub Zero should I do like a named topple just annoying where do I handle cast how do I handle cast we're gonna have to change okay that's fine that's fine here probably oh yeah okay cast D Type R um R Sub Zero it's a little annoying at the tunnel okay int has no attribute NP be an inch should be a d type couple I don't know I should get NP [Music] foreign is still passing in the old cast style that's ARG Sub Zero that's passing in falses now that's fine um that's getting the argument oh maybe this is the problem cool okay go back to this one now we want numpy to actually support bitcast [Music] how do we do a bitcast in numpy numpy okay binary representation of a number turns the binary representation of a number as a string uh sub okay oh casting unsafe okay so we just have to say casting this is nice okay casting equals uh no if y sub 1 else on the safe cannot cast array data from in 32 to float32 according to the rule no well that's too bad oh that's not what no means I see okay that's not a bit that's not a bitcast no there's another way to do this did they happen to figure it out in here how's it done enough CPU oh oh Dot View does that work okay that's cool try that yay wow this guy's good uh thanks again to uh Ben X we did a good job figuring this stuff up okay cool we'll call this Branch simple bitcast uh great okay uh on to the next one let's try torch doesn't work in torch let's see what this guy did for torch okay let's just view I'm a little upset that that's copy pasted um I think I should be able to just do this what um okay well those ones are stupid anyway torch doesn't support what foreign [Music] we're just going to skip these ones in torch torch really doesn't have these okay fine uh lvm right doesn't work at all uh GPU well this gets fancy and remember we're gonna need the linearizer they do not work why did you win oh you win 64 to 64. oh it is different oh that's cool that one is different interesting uh right so does that break yeah can I do or value error does that just work oh that's a little annoying that should really be checked for like way earlier where's the right place to check for that probably in lazy um okay oh oh that's just like a different problem okay this isn't actually checked anywhere um we want to check it in lazy dot pi uh search not argone or self.dype dot size make sense that shouldn't give me a value error though that should give me oh I don't know what's that right here bonus erases you shouldn't be able basically to cast it yeah okay something else is broken here did this never call cast no I mean it definitely called cast right investor fail all right sir not ARG one so there is our one or what no it doesn't make sense self.d type is float and the size is one the size should not be one oh sizes when you do item size damn it who wrote this crap no actually that makes sense cannot bidcasters G type item sizes work universally for all I'll just make sure that that actually catches the assert error here [Music] I'm not feeling pretty good about this now actually it's a test that's fine those look pretty good [Music] um okay it now works with CPU and torch so maybe what we should do now is um rewrite the state thing and see if llama2 works like this so instead of going to llvm I'm so excited to delete that oh we can delete this I wrote like a crappy thing in all of yam uh we should probably um I don't know make sure to be on CPU that should be free uh dot half to device default this is right now oh no because it doesn't know how to do that let's just see what happens well that's just not right at all because that's not actually the type so it must be somewhere I didn't uh what so okay I know what it is do I have where is can I like realize cast or something no but I probably have some special hacks for the realization of cash here well that's okay those are just hacks for image then all these casts here have to go let's see what other tests fell those will pass no no that doesn't AG types a d-type mismatch on Realize there must be some bug like I must handle cast in here somehow okay I move the cast around that's okay little self.realize Dot d-type yeah uh what sets that it's all fine right those are loaders there is somewhere else I talk about cast I remember I was looking at it before [Music] but it's there and that should be correct like out D type should be right something is setting realized okay where do we create the raw buffer okay this code needs to be refactored like this is hard for me to follow it's not too bad but like yeah these all these casts everywhere were such a mistake I should have never allowed this in you know like this is such a struggle like you want features uh but then you know features make things hard to read so we've got to spend more time refactoring how is this happening d-type mismatch on Realize buff up source to realize so it must be creating a lazy buffer uh I have one creation of it and it's here let's uncomment that isn't a shape anymore d-types half all right that seems fine this is where I think it would be messed up but it's not oh what could it be the lvm back no I don't know it's even the other Tessa family oh but I can't create a pull request because I'm not logged in great oh it's nicer if you ever tested fails I mean this might just be a b float 16 problem too I'm not sure how anything's really supposed to handle that oh I do have like a check for that none somewhere because I made a non-d-type because everything else kind of seems okay and all this poorly I wrote hacks I wrote hacks and this is what I deserve uh so if you look at helpers you'll see that this has none here because B float 16 is not supported in numpy so it could potentially be related to that I'm just trying to realize um oh I did do I did edit cast correctly right you can tensor.pi it's cast apply bitcast equals Joe bitcast equals true d-type bitcast input d-type self-depth cast that's right oh we got something to fail all right sweet well thanks for these very useful Trace backs no these just look like other weird failures it's because it doesn't download them anymore and they're expected to be like cache somehow why do I merge anything I'm going to be mad if that's what this is okay good never mind that's the good one sweet okay self dot cast d-type stuff from MP comma false okay um realize from it doesn't look fine oh I didn't say CPU equals one it doesn't matter okay there's a deeper bug than that uh oh I'll still some debug in there and see what debug tells us Tuple has no attribute name okay well no one tested this anymore oh Red Dot d-type is wrong okay somehow the d-type being assigned to stuff is wrong we'll just throw in a cert for that and we should be able to catch it create a lazy buffer I don't know assert uh his instance D type I wish there was a way to just run time type check python see maybe my pile just tell me what I'm doing wrong no so it's not there there must be something else like setting the d-type great this is a new computer so like nothing set up um come on my pie tell me what I did wrong okay so now that's becoming a topple where am I setting d-type you said it right there be immutable right we set it here that's okay oh we said it here okay so it must be this returning the wrong thing from exec AST oh do I have a Handler for cast here wow I wrote unreadable code I'm very sorry to everybody read this man because this isn't the right exact AST it's the other exact AST it's this one and then does this have a Handler for cast no more short lines this is unreadable if it's not short lines you get crazy amounts of code it must be this it must be from underlying or something is this had a Handler for cast oh here it is is it this no but that's probably wrong too you have to get the d-type which is right but it must be something similar to that I wonder if that actually fixes the GPU one let's get rid of debug equals four it's an hpu that's probably the same thing as a GPO yeah it's still wrong how did that not fix it [Music] maybe it did fix I just didn't save it right it's still wrong I mean but that made so much sense it makes so much sense that it's this because that's how it figures out what the new type is it does it in flop counter um like how do I change that it changed nothing it was like it didn't update the stuff Intense or also touch it I don't think so I mean there's a real question of how is numpy getting the type right so it's being called here in exec AST and then it calls this from underlying stuff yeah here okay from underlines you're turning the wrong thing from everyone is not even a thing it looks like so it's just a function is returning the wrong thing the ark the buffer is wrong oh okay from underlying is being called here with a raw numpy buffer from CPU this might be what's wrong this stuff needs to be refactored okay okay it's so hard to follow it's not that I don't know to be fair like it used to be easier I definitely know that what is that buffer class where's that buffer what class is that raw buffer no yeah it doesn't even make sense assert as instance D type D type and yet when I print them out right how does it make sense oh I overload and knit in some of these and then I never call Super you know what I call Super watch okay well we found the problem something they are is broken it's the cast foreign and then this is calling function for off oh okay stupid stupid I was on the wrong type never mind this is a very reasonable bug this is this is a very reasonable bug I wrote hacks I wrote hacks for the disk type and I forgot about the desktop all right well well that's what I get for writing hacks foreign actually I think this can enforce hey we want all of those to be bit casts actually because discs can only do bitcasts right now discs can do all sorts of weird casts I don't know this can do other sizes whatever that fixes it for now see I'm writing a hack so then I can complain about it later because otherwise what would I have complain about if I didn't write hacks now I'll delete that delete that okay well that was a reasonable bug actually um let me reset State and we shall try again it's good I'm glad we fixed it for that too that that's the cast there and that's the cast there I don't even know what calls that cast but whatever that's a little problem of course that cast whatever so the problem was and the the thing that we didn't test for properly was when we use it when we cast on a disc pointer we need better casts we need better tasks for that let's see what we get now we're back in broken lvm land um so what if I do this should bitch me out and tell me that it can't cast uh B flow 2 is not a valid type yeah numpy D type is required for gcpo we can improve that error actually um okay that's a nice hour right uh okay so we have a disc pointer so what we want to do is say rat Dot cast all right bitcast d-types dot oh I don't know if you went 16. do I have n16 I don't have any other stuff short you know what it's called a short no do I nobody does um let's remember these okay foreign d-types in 16 that's fine then yeah okay let me uh I wrote the thing that I wanted to do here okay it's this basically cool uh yeah so that's we have to go to CPU here should be able to do this then dot half 1.2 device default does numpy support plus 16 I forgot all right it's just those balls it works um does that work yeah loves being slow though we need tests for this big casts you're live on the bitcast remember you ate salsa soft shell tacos no shelves okay I'll settle for medium chips is this a negotiation you know have you guys ever thought about the time that Nelly robbed the jewelry store and told him to make me grills I have some theories that this didn't happen because like what are they gonna do like okay this is a stick up make me grills and then he's gonna come back like okay sir your grills will be ready to pick up in three days your grills will be ready to pick up in three days sir oh we should also check what llama 2 outputs when we use the boring prompt I have a sample prompt oh now it's slow now it loads even slower great but that is clean right I don't know but that's fully slow like it might be faster to put the half afterward well we'll see if the two things return the same thing I'm saying no I'm saying this to me know what I'm sayings whoa okay we have a new one here I'm a 20 year old girl who's looking for a good lay in Palm Coast wow this is the default wait so the default llama 2 prompt when you say hello they're both 20. [Laughter] lava two way better man no like he he the the Llama one guy goes on to talk about how he's like the member of like identity groups and stuff like I am a member of the black student union I am a member of the Hispanic Student Union 20 year old girl is looking for a good light in Palm Coast where's Palm Coast hmm yo guys you know the internet didn't always suck wow Florida oh you know man I'm on the other side of the country [Laughter] meadow knows what the people want man foreign thing maybe it'll be faster she goes to another school I mean I want to talk to her why don't I wish I can I turn that compiler warning off it's so stupid it's about includes or maybe there's a way to just fix the bug I'm 20 year old girl who's looking for a good land Palm Coast all right all right let's try the olive young one it should be the same I like wrote that code no I don't have to pass something into python is there a way to like is there a way I can just enable that and Pi opencl without rebuilding Pi opencl I looked at the code briefly and I didn't see why 20 year old girl who is looking for a good lay in Palm Coast great no you can't know one makes it more come out the coast we'll get together have a few laughs type beat crank up the temperature no no we test it with no temperature that one I'm pretty happy with that um I'm not trying weird fine-tuned 27 llama I like I like my base llamas well I did try a lot of 7db chat someone sent me a link and I was very happy with that the world's actually based it's just commies who try to like pervert it to not be you know can we can we realize that you know what I realized yesterday socialists aren't real they're just Communists in Disguise man like this just it's just Communists look up what socialism means a political economic theory of a social organization which Advocates that the means of production distribution exchange should be owned and regulated by the community as a whole uh basically we hate markets you know but I'm not anti-regulation I love regulation but only if the regulation makes the market more free we need markets to be as free as possible and sometimes you get assholes like Qualcomm uh who price discriminate and make life terrible for people because they're not regulated enough price discrimination should be illegal and then we can have the freest markets and everything will be great and the number will go up um I genuinely don't wait there's no Communists what everyone who says they're a socialist this is just a communist bro like you know you pull off the mask if so oh hey it's a communist oh a specter is haunting Europe the Specter of wow wow this is really getting this is really getting I don't even know if this is appropriate for twitch man this is this is really wow no we're clearing it bro we're clearing it and the rant about socialists all right that's it guys it was it was nice being a twitch streamer um no that is not true that a regulated market is not free by definition no this is not true you need to regulate the market to make it free right you do not want monopolies and you do not want collusion right because because if you have it yeah okay well I'm not gonna go into this but you need a regulated market but it needs to be regulated to be more free because socialism lost doesn't mean it doesn't work you mean communism we need to stop saying socialism it's like a euphemism you know what I mean it's like it's like a euphemism because like everyone knows how terrible Communists are so they'll just don't worry guys it's not ecstasy anymore it's just clean Molly um as a 20 year old girl from Palm Springs I cannot help you overthrow the government oh man we're not talking about those government man Communists you know nobody how do you prevent regulatory capture because you just make the regulations not the kind of things that are being regulated now right you don't regulate something in order to make it safe you regulate it to make it free right if you're regulated to make it free you'll get everything else as an outcome right like I don't know dude do you you was poor bro it's sad it's really sad I was over there last year it's like it's sad you just look at there hasn't been growth in these countries and like you have a degrowth strategy and that's all great but like you know like like look at okay all right I don't know how we started on this um no the Nordic countries are fine actually the Nordic countries don't seem to have this problem and I mean we can go into theories about why no it's not diversity in the US is legitimately great I I legitimately like diversity guys like there's a reason I live in America you know um it's it's just a shame that like they weaponize it's a shame you know that's what I mean like it's a shame that like like these like racists found this new way of just like we're just gonna scream diversity a lot and then we can be racists no no you can't racism sucks bro but if we scream diversity and talk about inclusion then we can be racist and no one will figure it out no one will figure it out when we put the whites on one bus and the blacks on another bus we'll call it I don't know what we'll call up we'll come up with a good name for it and we'll say it's under the banner of diversity and inclusion right because if we say it's under the banner of segregation and racism that's bad but we want the same policies so we'll call them diversity and inclusion and then we can put the whites on this bus and the blacks on this bus which is what we wanted but no you're fucking racist and you deserve whatever hell racists go to right does it make sense that's right that's right you can go on we have two buses and there's the expense bus and the cheap bus and that is how we discriminate against people based on how much money they have and if you want more money you can just work harder or we can have communism where there's two bosses and they both suck [Laughter] people just just not like no but I'm not a racist fan these people we have these people he's raised in this Zone and that's not racist because because I got a degree in whatever man I got a pretty impression studies there's two buses and they're both going to work camps I love it I love it [Laughter] um no they'll never be abundance abundance is a myth people will always just invent new things to want you know when you have to say I am not racist you already lost yeah I don't know none of people have called them racist I don't know how they really respond to it like I don't understand why these people genuinely just didn't right away like like it took a bit for everyone to kind of see through it you know what I mean it's like one of them Rap Battle Tactics you know you know when you like like like like do this like whole whole rap battle tactic about like you know like like loudly proclaiming like like how large your dick is you know what I mean like we know it's not big bro like we know it's not we know it's not we're I'll call your bluff whip it out like just like oh you're late to the party well welcome to the party um we are we are streaming and we believe that zones that only allow One race of people are racist that's what we believe on this stream um if we discriminate against people based on their race that is definition textbook race 20 year old girl from Palm Coast yo mama two you were very inappropriate you were very inappropriate on my twitch stream and you get a strike um okay we got to finish doing we gotta finish doing bitcast oh we have the meeting okay so every unless it's women-only clubs yeah I mean look that's not racist at least uh you know the the the race is diverge genetically 20 000 years ago men and women diverge genetically a long-ass time oh foreign true facts true facts how many genders are there I don't know whatever gender you want to be you can be see and that's that's the flip side of all this stuff like whatever man you want to change genders you want to be a different gender you want to say you're a different gender like that's all cool man like whatever you know who cares and that's the problem that's the problem both sides have agreed they care and that is how they get you the only way to win is not to care I want to be software too man someday I will be software someday I will be software and someday we will understand the Socialists such as communists all right why do I ever turn off subscriber only chat and and let you no no no no no no no no no no one should Target your children no one should Target your children this is not this is not okay but see that's like this is what I mean like like you can't you know there's just no Nuance in society man there's not no Nuance in society and that's the problem because if it was nuance CNN wouldn't exist and we all just live to serve CNN oh let them divide you and get over it thank you thank you um yeah but yeah I'll repeat the live I heard Barbie described as post-woke and I really like that like guys it's over man the wokism's over you know what I mean I heard like after Mao died in China everyone was just like looking around and be like yo so that that guy was crazy right like he was crazy right everyone we all thought that right and everyone's like I'm glad I'm glad I'm glad you brought it off you know I'm glad uh I'm glad someone's saying it you know [Music] it's so good thank you all right so we go on Discord at 5 00 PM uh you are welcome to uh join the meeting um people might not want to talk today we'll see if people want to talk today because they are on stream uh let me check my DM over here um shouldn't be getting a DM about how tiny three is down put down uh let's see what tiny bit is down I got a DM about how Tony threw his down all right so streamer Mode's enabled I don't know what streamer mode is but we're joining tiny voice alright so we joined tinyvoice uh every day at 5 00 pm uh and we talk and I do it with headphones because otherwise there's Echo tiny 2 is off uh tiny theory is up so I don't think other people will be able to hear the other side of the voice chat uh oh this is hard to do on stream also oh this is Sky Brave you can turn Sky Breeze Discord she streams on Twitch um someday I will be friends with skyberry all right so usually what we do is we go to the tiny grad pull requests and decide which ones we want to merge um but yeah no I I don't I use a different browser for all my streaming stuff so uh I've not actually logged into anything um okay support symbolic expands Ford expanded to symbolic shape by doing the bounds math this is probably fine to merge oh I guess this is fine okay so this is a little annoying because LT nodes can have both uh can have two nodes if self equals B return num node zero wait how did this used to return false I don't really understand this oh okay uh if South equals B then yeah obviously I guess less than is not true something's not less than itself so I guess that's fine yeah I think this is fine to merge uh I found with merge in that uh for this yeah you can merge it yeah um for this one I'm reluctant to merge things like this that like add lines and it's not clear why is the creator of this pull request in the channel uh okay I don't see yeah I like I ran the fuzzer a few times and it didn't find anything the risk of bugs in these things is like kind of high or like is that always true pulling a gcd out you know see there's a lot of subtle ways for things like this to be broken and if it's not it's not like benefiting something I think it's not worth it um yeah you will have to check those I think we I think we closed this pull request merge close this one's work in progress uh think it's quite enough to lock the Bounty yet so I I'll show you guys the the Bounty spreadsheet um I unlocked a whole bunch of the bounties yesterday just ones that were like stale like you want the Bounty bring it over the Finish Line um in order to like in order you get like zero percent of the value for eighty percent of the work it's not until stuff's merged the problem is when things aren't merged they become stale um so there's good ways like all these are kind of in like a kind of half done state if you look in the closed pull requests uh so you know there's there's it's pretty easy to go do these penalties uh literally free money um this one is here supporting this bounty it's one of the thousand dollar bounties uh a hard one um this one no way yes legit Obama uh uh why do I have a different name here oh you changed your GitHub ID okay cool I'm not crazy that's good um yeah again I don't understand I think that should the test failure is unrelated to this I know it's a linter failure okay no excuse for that um this good to merger now okay uh that will be cleaned up I'm echoing through somebody who am I at going through uh oh and then this one no no found what's wrong with SD Okay so this one is a deceptively easy slash hard four hundred dollars um it's four hundred dollars and it's very few lines of code but they need to be much better tested like now that I see it we merged this once and then have to revert it because it was broken what this basically does is when you're doing a reduce when you're reducing like a big Matrix down to a single number it breaks the reduce into two parts so you can do first The Matrix down to like a small Matrix and then all the way down to one and it lets you get full utilization GPU um yeah okay so I'm I'm scared to merge that one again uh that's work in progress uh g1y how we doing on this guy what uh what what what what what loss are we up to what um there's a bounty for getting the loss on cfar to 94 which matches so there's this thing called a C4 hlb um before we can like sell the Tiny Box I I want to beat this so I want the Tiny Box to basically be better than na100 um and it's the same price as an a100 so 80 gig a 100 is going to set you back about 15 grand and that doesn't include the computer so it has 80 gigs of RAM so tiny box has uh 144. and it has more than double the flops uh like triple the ram bandwidth so we should be able to beat an a100 comfortably uh on cfar and then all these bounties here are for training all the ml perf models it's on my board over there AMD on mlperf cool can I lock the SVC boundary yes I can this one here uh again so I'm not I'm not logged in I'll do it over here you know I need an AI to maintain the spreadsheet for me Jacob spawn is now locked uh and I will change least compressible to chenu XYZ [Music] got it uh yeah I wonder how much the the LR thing helped what else oh weight normalization like what else are we missing from the original repo [Music] you took wait we're not using value we should be able to use value if you if you're still getting 94 on the main repo putting these things back then it doesn't matter yeah I mean I'm fine with like like it's I think that's a smart way to probably approach it like take the main repo and just rip things out until you still get 94 and then kind of meet in the middle cool um yeah this one's exciting and then this one's exciting uh training multi-gpu uh how close are we to merging the first half of that cool uh yeah okay [Music] oh wait but this doesn't require oh I see okay but this one doesn't use P2P I see I think the main thing I think what we're gonna have to end up doing I said this yesterday is rewriting AMD back end um after this is merged I can I can start working on that I mean we have to do it I think before we get to uh uh like seriously training these endless models uh at these uh sorry these animal perf models I think there's two big things still like missing um one is we got to get like the gem stuff integrated like we got to make the single GPU faster and then we got to replace the back end with something that will run this stuff blazingly fast um like a Cuda cash looking thing I don't know if if hip has one but I want to try to like just get to the kernel and and talk to the colonel in the uh so I have I think it's in tiny grad um an extra Rock M yeah I have this thing called sniffer uh and sniffer just shows you all the actual eye octals so I think it's going to be like 500 lines of City um to talk directly into the GPO and then we can like use the D imagine if we want or to choose anything wow and this is all separate from which compiler we end up using right so this is like like it's a little bit overloaded now but there's a distinction between runtimes and compilers um so like I think maybe I don't even like I took this all out uh but live here like the code gen if you have PTX go Gen kudoko gen and Triton and they all generate PTX code which is done in this uh in this runtime and see this even still includes a compiler which should probably kind of be taken out of here that it's just the runtime not the compiler if that's doable for the thing cool anything else sure uh did I see it got it um well the other one okay yeah I mean what what I need to see is like here what's specified in the uh cleanup code support all sizes test with eval sat so I want to basically see all the sizes and then the eval set and making sure that that scores as well as the open AI one like that that was the whole things went off the rails like that was the whole boundary from the beginning the whole Bounty from the beginning was all sizes test with email set bye well cool let's uh yeah post the post the the results like we did for the the llama uh the level the level one above it did a good job of like posting the like standard LM of valve stuff um you know we got to make sure that these models are not just implemented but implemented correctly yeah removing removing the Bros is great cool um yeah I mean is it just needed in the pre-processing or is it needed in the runtime if it's yeah I mean if it's needed in the runtime we have to do it but if it's just needing the pre-processing I don't know but I'm happy to rip it out even if it's like 100 lines of code or something wait what do you mean it's five dependencies or more no no no but what you're saying is librosa how many lines would it take to replace the brosis what I'm saying yeah yeah great answer place it uh cool yeah I mean yeah but but for that pull request what I need to see is all the different sizes versus all the different eval scores uh what TV do you have oh that's pretty old ah is it slow is it do you wanna do you want a 3900 700 I mean oh well it sounds like we got to fix that first I mean this is this is part of cleaning it up right and this is what I mean about like why I like having the uh why I think like like the goal of the bounty is to have all the evals run because you got some GPU and we're able to run it on them right like it being too slow is a problem okay so you say you're saying it's blocked on on jit llama to be no but you see this shouldn't really be a problem right like if you can do it for one size the jit should not make it any like like the big one shouldn't be any harder than the small ones right because that python overhead is constant regardless of the size of the model well actually that might not be true if the model has more layers that's not necessarily true but if just the things are bigger it doesn't matter that's all GPU but cool yeah I mean post of numbers about how slow it is and how long the test that'll take to run then and then we'll see what to do from there cool uh 22 23 is back up let's see if tiny threes back up 23 is back I don't know why it went out did someone do a pseudo power off or maybe I'd kick the wire yo yo um so we're over here on Tiny five uh tiny five is the latest in Tiny Box technology it's right over there um yo I was working on I finished the cab bottle for the case well I finished the outline of the camera off of the case we're having comma's mechanical engineering team actually do the case um I'm uh yeah I'm quite excited about how it looks and then we're gonna have tiny boxes in commas data commas compute cluster and then we can uh not have to power limit them which will be nice at my house they have to be power limited cool anything else for the meeting great oh oh unified attention I would love to merge that I would love to merge I want the good torch one not the not multi-head attention but the torch one that's like like I want to add that to tensor I think it's like qkv thing yeah yeah yeah yeah scale dot product detention yes that's exactly what I want in tensor and then uh yes let's let's write that and then we can put the flash attention bounty on I mean yeah so okay get it to the point where llamajit's not merged uh and then if you get the pull request to a point where you say this is how much speed up I know we're gonna get or you know what you can even do just just uh merge in I mean pull in llamajit stuff uh if you can do that or or get it to the point where you're like look here's the eval set I'm only running on five percent of it now right like five percent of videos that's already pretty good I think that's what the LM one ended up being or a similar problem so that's totally fine what's a flat astn compiler I mean we have a linearizer I would I would put this in an issue uh if you want like like if there's some actual like Improvement proposal I would put it in an issue and then I can look it over there yeah look it's hard for me to think about like this put it in an issue and I'll take a look at it okay cool oh no worries uh yeah no and then we can like track it and then the problem with like doing it in these conversations and I need to be better about this too we track things in issues so then when someone else has the same idea we can call back the same conversation and this is ideally how it's all supposed to work one day llms are going to index the world it's gonna be great cool improving compile speed I mean oh I mean yeah it's great but how do you want to track it so like benchmarking things the CI is annoying and then doing uh uh doing doing self-hosted Runners is annoying is it a good idea yes all things are good ideas but what the implementation is what matters right and this is this is true for this is universal advice good for the stream too um with all things tracking stuff is a great idea now how are we going to track it what is the implementation of that look like what are the downsides right I have another saying I say I come all the time engineering is all about trade-offs right so do we want to track tropical time yeah that'd be awesome now is this tracking going to be useful is it going to be intrusive is it going to be like what are the downsides right got to be an issue gotta be an issue these meetings are sync up meetings uh yeah okay all right I'm done good meeting all right the question policy still applies in meetings like it's just it's just like we're doing something clear in the meeting if you want to ask generic questions there's other places for it it's not the meeting uh that's the meeting I hate meetings uh we do that every day at five uh you know like look the tiny Corps it's all right don't you know I don't know why it's talking about a company and like now I gotta finish this stuff I have money from investors I feel like a big deal like oh you give the money back yeah you give the money back but then you know there's a lot of money in tiny boxes there's legit a lot of money in tiny boxes um I hate managing people I really hate managing people like everyone needs to read the questions document you know someone in voice chat was just asking me questions like don't ask me fucking questions like like here's a questions document all right this is the how to become a hacker but there's the uh how to ask questions the smart way all right yes the smart questions document everyone should read this right um I don't know I kind of hate doing the voice chat meetings I don't know if they're useful we might just not do them um there's like I have something to do at five every day I mean it's kind of good it syncs things you know this is the like this is welcome to like the tyranny of remote work right like you're gonna end up with um you know what you're going to end up with it's such a double-edged sword what you're going to end up with is better and better