George Hotz | Programming | cherry computer: vectors and tinygrad | Science & Technology

Key Takeaways

hello twitchies hello bruce wayne 22. i did the math for you uh wait why didn't this open in preview why'd that open in chrome here it is in preview does it really they didn't put any index okay they know they know they know is that a drake lyric they know they know they know the real is on the rise fuck the other side i even gave them a chance to decide i just tell them they know no cherry computers not cancelled no no no look guys if starting your company is not a struggle then you didn't think about it enough when i started karma i was smoking weed every day real fat um so yeah we got a few things to do today i've been working on we can put the we can do the website do you guys want to do the website for cherry computer i was just thinking it would be that logo there's some guy linked yesterday no what's this oh floating points no no no i just think it would be that logo and then a picture of an nvidia graphics card and it would say don't you wish you had a choice all right well we'll just should we put the website together just because we're here having a good time all right well you know someone probably copyrighted this cherry did anyone copyright this cherry nah i can't be copyrighted it's illegal to copyright a cherry image cherry.svg we want everything of course to be in the center you guys know this is my this is my li html code what's that it's a thesaurus line d source battle html code no yeah yeah yeah oh yo here's the difference between you and me i do shows get lots ahead you do shows with ostrich head yo i do tours get paid and stay at the sheridan you give tours at the monterey bay aquarium ah yeah [Music] so let this skinny dork go back to making shitty mongolians if you ain't nailing a railroad you're playing yourself go bro and i don't mean he's afraid when i say that he's yellow here's an antenna tape and some velcro go make me a cell phone and do my fucking html code all right um so you know we're just gonna put it big yeah center tag better work don't you wish oh no no it's gotta be a picture wish you had a choice let's get a picture let's get a picture of an nvidia whoa css whoa who needs css well the wheels on the rise fuck them other side you know no no let's get a picture of an a100 oh that's not a good picture that's a good picture yeah i'm stealing this picture from my website all right i did hear that you could buy pictures of these cards on ebay and they were selling for a lot of money just picture all right let's rename that to 3090 image source 3090.jpg let's terminate my image tags do i know html we're not closing any other tags cause fuck closing tags all right um no no no we need some burrs in here yo where's icy narco yo we need some burrs burr we need more burrs all right we need we need more burrs all right don't you wish you had a choice now let's just put a lot of burrs here sherry coming 2022 we need some burrs over here all right oh it's got to be big everything's got to be big i don't know i don't know why you'd ever want your text to not be big we need more bigs don't you wish you had oh but now the burrs get bigger that's a little much all right cherry computer another bur here all right good thing we hired a graphic designer [Laughter] uh wait let's just copy this amount of burrs down here oh yes beautiful you guys like my website oh you can't see i'm in the way i'll put it over here so you guys can see all right it's a picture of an nvidia graphics card and it says don't you wish you had a choice and then it's cherry computer 2022 um for you want job only github and we'll put that in a comment so you have to be leave to find it perfect you know what it used to say on the comma website till everyone who doubted me is asking for forgiveness if you ain't been a part of it at least you got to witness bitches and that's how i feel that's how i feel waymo that's how i feel cruz you know that's all everyone believes that everyone believes it's sorry drake i stole your bars drake used to be good guys drake used to be good cherry's just the name of the computer you know we'll put it it'll be a good logo on the chip um don't you wish you had a choice cherry computer uh you know what i won't say coming 22 winning ml per in 2022 what do you think a parallel in 2022 no i don't think we can win for a while we'll just say we'll just say we'll just say showing up you're winning winning is winning is a little much uh winning winning 2022 on mlperf winning ml perfect no no no just just yeah yeah yeah yeah cherry computer custom card you can buy mlperf2022 find us on mlperf 2022 no we'll just be on ml perf and maybe we should link to that uh i linked to it already so i'll show you guys i have a business model i've been i've been working on i've been working on this so this is like the plan here we go let's say by the end of the year let's let's we'll be on mlperf oh we need late to be all capitalized i forgot cherry computer is an all capital company all right don't you wish you had a choice no and we gotta we gotta we gotta keep it with the you know don't you wish you had a choice yeah perfect cherry computer custom card you goodbye yo yo this is a good website let me move my let me move my face so you guys can really see my website i worked hard on this website boys don't you wish you had a choice jerry computer custom card you can buy where's the buy it now button should we take pre-orders but they're only a dollar like cyber truck pre-orders but they're only in cheap no i'm kind of let me tell you guys why i'm done with cheapies crypto is a cesspool crypto is a cesspool you know i was at lunch today and what we're just kind of saying is like who's creating value in the world you know what i mean you know something is a security when you're trying to make value off the air what let's let's look it up let's let's look up the howie test you know for all you guys out there trying to make shit coins and like you learn about the howie test right and it's like [Music] ah yes an investment contract exists if there is an investment of money in a common enterprise with a reasonable expectation of profits to be derived from the efforts of others now what you can find is that if everybody in the world is just trying to make profits derived off the efforts of others but nobody's actually doing any effort then everything's fake like imagine instead of the stock market being backed by companies because well they're really even companies anymore they're all just financial instruments right so it's like it's like you're just betting on all these different shit coins then you realize we already have this and it's crypto right like yeah so um yeah if you're interested in working at cherry computer um you contribute on github yeah all right i think i think this is i think this is a great website well it's just everybody today everybody today is like oh i'm just going to make money passively by investing off the backs of others but if no one's doing any real work like nobody has jobs you know you like go out and there's just people everywhere because nobody has fucking jobs anymore because you gave them free money and you can't do that so like welcome to the decline yo guys i was on um um i was on 4chan last night and i read this post and it was just like bros you know like all that like accelerationism where you think like there's gonna be like a big event there's gonna be like a big recession no it's not gonna be like that there's just gonna be slow and steady decline for the next 20 years and that's what i'm really scared of and then the third post now was like bro yeah read a history textbook that's what it looks like when you live in declining empires like it's not cool it just sucks and it just sucks slightly more every year and i'm like holy shit that's what we're living through yeah oh my god yeah ubi yeah let's all make money what if we all made money off of everybody else's work right this is all this every time you hear words like collaboration it's just all this bullshit it's people just trying to make money off the backs of others and that's great if you have a small defector class that scams everybody like that but if your whole economy is based on these kind of scams well what's left do you know what i mean don't you know no i mean we just need to legalize like heroin and shit and like all the people who want to like go do heroin can go to heroin and like you know all the people who want to wire themselves let them wirehead themselves don't give them ubi give them a morphine drip you know we can synthesize morphine in huge quantities um is this d did matthew mcconaughey say this in waffle wall street let's let's let's watch this clip this isn't copyright infringement because we're only watching a short clip no literally send the gme people to the moon yeah no i'd be like oh yeah the fugazi scene yeah yeah again it's all fugazi look look the gme people i don't blame the gme people but like does anybody think the price of gme has anything to do with gamestop oh yes remember when gamestop launched that great new thing but then it turned out to not be great but then it wait no gamestop didn't do shit this is all just speculators right and then like all these people are like oh yes oh yes oh why a crash oh yes market watch yes vox please oh yes the motley fool let me guess it's gamestop right like i hate all these people man it's not like it's not you know it's not the guy who like like memes gamestop who i hate right it's anybody who takes the system seriously recursive speculation it's not a short squeeze anymore it stopped being a short squeeze wall street's making money on gamestop too you know who loses money everyone who says diamond hands diamond hands diamond hands like you're gonna lose it's a question of when like like the market will be eventually look i don't want to talk about this i'm just saying that you know we're going to create a new society where people who want to create value just for the sake of creating value i decide that i'm going to create value because it pleases god and like it's that simple and you can't really have like another explanation for it right like it's like there's no reason to create value today i can make bajillions of dollars scamming people not bajillions but you know i could have i could have i could have a solid you know like 50 million dollars just off scams maybe even more um that'll work the system fuck the system man all right you know what we're gonna read comments and then we're gonna get into today's stream um i saw i saw a comment i saw it was a good comment i liked it all right haas is depressingly cool anyone who types louder i mean george is the example of how you can act if you're successful and you do moral honest work you know what's gonna fix this guys shaming when somebody like makes money off the backs of others you should shame them when somebody like collects unemployment you should shame them right like this is the only way we're gonna fix this uh you know what's what's that um it's tupac man two pocket yeah you gotta operate the easy way i made a g today but you made it in the sleazy way yeah yeah um no not social credit like this thing you can't do this top down you got to do this bottom up like every one of you we got 420 people out there yeah 420 420 blaze it uh you gotta just like you know like that's not cool bro like we all just got to turn to people like you know unlike what everyone's like being shamed for like you know publicly on twitter like fuck all that noise i mean like go to people personally and be like yo like you're a loser i like i've done this to people you know like this is just you know you're a loser man like yeah okay yeah make money off the backs of others who do you think built society you know um oh and like i do this to these people like i i meet these people who you know create no value they actually they just take value from the system and like just you know bro like you're a loser well no we're doing a malpurf i'm not even talking about like the people buying and selling scams i'm really just talking about the whole i want you to you know wouldn't you feel better making as much money as you can so you can do good things with it no money is worthless money is backed by human effort if there's nobody putting in any effort money can't buy you anything why is the inflation so crazy it's not because of the money they printed it's because you have people now who think it's an okay lifestyle to not work and like no you're a piece of shit right and like this is what we need to bring back do i think everybody has a moral obligation to contribute to society i mean depends what you mean by society yes i think everybody has a moral obligation to be a positive force in the world of course yeah elon has more tolerance for this than i do it's just it's hard for me to tolerate um no i think elon creates immense value yeah i mean that's what exactly like elon lowers the cost of many systems right like think of the boring company it's just like that's what we need more of in the world it's like we're going to take this thing that we can do we're going to make it 10x cheaper that's what makes the world better um no it's not about incentives and this is how we're going to move past like this fake capitalism shit like like like you know you just gotta look at people and you know it's kind of like what people say about like the billionaires like do you really need more money you know um well yeah people who put effort into worthless ventures like i said this stuff a lot on my stream but this is really like like a call to action if you work at a job where you know that you create no value quit quit or you're a total loser right you know like i'm sure some of you i'm sure some of you out there have jobs and you know that you do nothing you know you contribute no value to the world you might even take value from the world quit billionaires reinvest in like billionaires aren't necessarily a bad thing i really don't think a lot of this problem is is is like a lot of a lot of billionaires get an unjust amount of hate for for what they they do right like most billionaires are new money and they most people today who are billionaires are billionaires because they created immense amounts of value i'm not talking about jeff bezos i'm not talking about mark zuckerberg i'm talking about all the people in the middle i'm not gonna like i'm not making a moral judgment of of tick tock right like is tick tock as a company negative value i don't know i think people enjoy tick tock this isn't what i mean right i mean the people who you contribute nothing the people who contribute nothing to society the people who society would actually be better off if they didn't exist right the people who are just attacks like i'm specifically talking here you know a lot about all financial services right hedge funds you guys are scammers like it's not okay and like like we have to we have to like get to a point where you know we kind of say this right like like you're not you're you're you're you're not you're not setting the prices of things right like that's a lie i'm not saying facebook's crap i'm not saying facebook's crap i'm not saying tick tock scrap those things at least look like it may be a shitty product but they've at least kind of built a product for people i think they have shitty business models and i think there's problems with them and then you know what we'll even go like like like hedge funds particularly you know that they've taken up all these smart people and they contribute no value to society um yeah you know facebook has an ai research lab and like honestly i think you know what here's what i'll say about facebook and tick tock right like they're somewhat addictive but people do spend a lot of time on them and people might say they don't enjoy that time but like i much prefer to look at revealed preference than to look at uh you know stated preference i'm not talking about the things you know about i'm talking about every single person who works in administration right i'm talking about every single middle manager out there i'm talking about every person who works in like finance industries that have nothing at all to do with people that provide no value at all to people no value at all to society and are just a tax you know in london they pay us way too much but houses are also expensive when you all hate our jobs quit quit you know you look at go figure out how to produce value and then also you have a whole class of people who i have no idea how they get money but they're perpetual travelers right this is another you know what the difference is between children and adults children consume and adults create these people are grown-ass children i don't mind poker players look look poker players don't take money from me the problem if hedge funds only competed against other hedge funds i'd have no problem with hedge funds it's just a massive tax on the whole economy the whole financial industry's attacks on the economy poker players gamble against each other that's great that's great you know you know what i'm not yeah bankers get 120k out of university it's just too huge to refuse it's fucking fake i don't know who scammed you into equating like these numbers with with any kind of like real value at all the money is made up the money is backed by nothing you're playing for points in some guy's game and the guy who made the game rigged the whole thing money doesn't get you girls that's the dumbest shit i've ever heard people who think that happens have zero luck with girls you know what gets you girls man talking to fucking girls it's really not harder than that money doesn't get you girls anywhere this is the dumbest myth you know who promotes this like club promoters like like like this is this is this is just it's just to promote no one gets laid from clubs man you know no one's like like like i went home i met this girl at the club and i brought her home this is rappers and shit lying to you you know and it's like oh yeah the club is just a way to scam men out of money oh my god all right i wasn't expecting ranting some women will go with any guy that has money wait you mean like hookers i mean hookers aren't that expensive guys like you know there's there's there's great hookers on the internet and like it's not that expensive right like we're not talking like lots of money here i think what you're confusing uh i think you're confusing money and status right like yes women you know you use your women to like status there's like you know seven million ways to have status and your blue check on instagram will do you more than money for women yeah no oh my god in cells well are you allowed to talk about intel's like insults are the most self-loathing little fucks like if you guys just went you know back in the day 10 years ago like a pickup used to be this like great thing like it was like those things like yo how do you get laid man and like it turns out there's like ways to do it and the biggest piece of advice is go out and talk to women right and now you have these like like in cells who are like i'm this or i'm that or women won't like me because this no women won't like you because you're a little bitch who stays on the internet and talks about how women don't like you yo i own 24 30 90s giga chad right here women will never like you if you go on the internet and complain about how you can't get laid i'm gonna put a title on this webpage let's put a title on this website cherry computer you know does somebody tell you guys this stuff fucking victimology man you guys wanna still this is because it's because i'm only five seven and there's guys out there who are sixteen shut up you know all right what am i doing closing tags well i kind of want to close those tags because then the whole thing is going to be the title in the head all right um all right we're done with the rant what do you mean we have to talk to people it's rigged man i get out uh you know picking up women is is actually one of those like it's not rigged and it can't really be rigged because it's very decentralized right there's like lots of women out there people like oh well this this and that on tinder well yeah don't meet women on tinder man like that's a tinder problem oh it's it's in you know oh man you got you guys know it's my favorite um how to stop failing at life quit you know you you guys uh bitching about the bitches edition um everybody everybody everybody should really read and internalize this i'm telling you like i i found this when i was like 15 and this was like one of one of the the the most uh you know the um go where the girls are what you know should go out clubbing do you want to date a holla batgirl fuck no then no don't go out clubbing you may not realize it but fully half the population of this planet is female you don't have to suck up to an asinine subculture that you don't actually appreciate just because there's some single girls there there's single girls everywhere just do something you enjoy in public skateboarding warhammer 40k whatever i guarantee there will be at least one female there eventually you know what did lord say maybe the internet raised us no it's not cherry ai it's a cherry computer we're building computers for the future all right wow i get too amped up from the rants you know we have two streams on twitch what are hot tub streams i heard about this i got an email from twitch talking about some new guidelines and it's am i allowed to show nipple and that's the only thing that i was interested in the answer is only if in a nipple appropriate setting all right so this is the risk five all right maybe we gotta do meditation you know it's been a while and i believe there's a time for meditation in cathedrals of our own nippler they teach this shit in high school the schools are propaganda machines run by the government you guys like you know we had like all this propaganda during like world war ii and during the cold war and all that kind of shit you know it never stopped right you know you know the propaganda never stopped you just don't see it today because you call it normal but you know there's a lot of shit that's uh propaganda man and that's why they don't you know teach you that in in high school um now the the all right exercise six finding your soundscape all right we got we got a signal quality check on here no no we're gonna we're gonna meditate um hmm i think i need to get a i think we need to wipe the sensors with a damp a lightly damp cloth i dated this girl she's homeschooled uh now she like went to a bunch of different schools she's homeschooled like you know it's just crazy when you meet people who like weren't brainwashed like this girl's like a genius and could think like clearly about society you know try to get the ah ah here we go for this calibration find a comfortable position and close your eyes take a deep breath muse is now listening to your brain signals relax and let your mind flow naturally let any thoughts feelings or sensations simply pass without any kind of judgment [Music] welcome back finding a comfortable quiet place to practice can make a big difference with muse it's just as important to find the right soundscape to keep you on track our relationship to specific sounds can influence how we feel during a session this is why muse has different soundscapes for you to choose from let's explore the soundscape you've chosen for this session when you're ready get comfortable and release any tension you might be holding in your face and jaw for this session whenever you notice the sound get more intense instead of reacting immediately stop for a second and listen try to hear the weather as clearly as possible notice how intense it has become and be with it for a few seconds then let go of the sound and come back to your breath ready place your attention on your breath now great job that was not my best one uh i did okay but the important thing is we're calm um oh i have something else for you guys you guys know about comic-con we're gonna have to all get on topic a little bit i love wait the waking up app's great uh this is a muse i did like all the good waking ups um well i've been out of practice with meditation lately meditation's not bs does the headband bs yeah i mean it definitely correlates i don't know i thought it was a cool product so i bought it i mean i know i'm a lot more calm right now uh yeah comic-con yo so we got we got two tickets we got general admission and we got vip um it's going to be good it's we're putting a lot of content together to kind of present what comma has been doing for the last five years um you know open pilot is the culmination of tons and tons and tons of work and i'm hoping by kind of putting it together like this a lot of other people working on self-driving cars will kind of realize how hopelessly uh behind they are if they watch this kind of stuff um after trying tesla full self-driving uh i've become pretty confident khan was gonna win um you know part of the reason i'm doing this is self-driving cars are still 10 years away like true self-driving cars um you know i'm talking like level five uh i think i think we might just see them by the end of the decade but the current amount of compute uh available is not big enough to drive i don't think uh so one of the goals with uh you know cherry is to build accelerators for calm right now comma uh our devices uh have about as much inference they're they're about on par with a b and then our trainer is about on par with a mouse i think in order to drive you're gonna have to get inference on par with a mouse and a trainer on par with a human so i went over the cherry been working on the cherry business plan a bit here um charge design with need in mind we upload small risk platforms into the eye cache and let them run uh so i think we've answered both of these questions i've been texting with what referring as the source um and uh some great ideas about really how to build this thing in such a way that it won't just be fast for the current neural networks but it'll be enjoyable to program sometime you'll program stuff and think of like what's enjoyable to program and think of what's not enjoyable to program right like you know does anybody find writing multi-threaded multi-process things to be fun no like that's not fun what's fun like like code golf code golf is fun um is cherry now a serious thing i mean let's see so here's comma comma very serious um here's cheapy and cherry's somewhere in between uh elf need need is uh it's a thing an open pilot to accelerate models you guys can can look it's one of the other ways that we beat there video so our first step is to build everything in this board um with a four by four by four manhole unit uh so you know it's it's this is only going to be at 50 megahertz we're only talking 6.4 gigaflops here um this isn't going to be serious but we also need to get forward backward passes of these three nets are the three that we're targeting um resnet50 and bert large are in mlperf uh i took mlperf more seriously actually i like mlperf a lot there's only three accelerators listed on mlperf right now and only one of which you can buy uh there's the nvidia which is one you can buy there's google which you can only rent and then there's huawei which i don't think they're allowed to make any more of because of the tsmc thing um so we'll be able to train mnist models right on here and we'll see if we want to use this ddr ram uh on here as well to like that'll be the big address space that we can we can dma into uh and then after we've trained mnist on here hopefully in a few weeks we'll buy um alveo u250s and a linux computer and then we can move to we'll put a link computer in the house um so this is not going to have obviously very tiny i'm going to get the dma to work over this ethernet interface i think it'll be easy on here we'll support dma over pcie we'll have a 20 meg on board sram we also have 64 gigs of ddr4 and we're talking a 16 by 16 by 16 map mall engine and that will be four teraflops um and then on this card i think this is where we write like the pie torch port um this is where we make multi-card training work and i'm hoping by the end of here without even taping out a chip will have something that's decently competitive on mlperf with um maybe not a100s but maybe like 10 1080s or something and then when we do the the the tape out of the cherry two this if we actually take this chip out i'm hoping to do it from like a turnkey place so i can just hand them the verilog uh we do a tape out i want to sell these cards uh for a thousand bucks and then um i'm hoping that even with a lower power budget uh we should be on par with a 30 90 and one of the reasons we can do this is there's a few things about the 39d right so the 3090 is a massive multi-core machine the 3090 has 6144 cores all right the cherry two has one core but that one core has more teraflops than the 30 90. so we're spending a lot less power on dispatch we're also spending a lot less power on copying back and forth to ram um so our cash is a good bit bigger not the a100 has a bigger cash actually we might want to up this to 16 million elements because then we're on par with the a100 cache [Music] and then yeah if we're here and we've sold 10 000 of these cards uh we're on track to winning the ai chip market at this point we raise a large amount of money um we hire people who actually have expertise in taping out an experience taping out high process nodes so in 12 nanometer i'm hoping yeah we can just bring it to a turnkey place hand them to the verilog and be like yo uh do the best you can to lay my chip out um but after the cherry two uh if this really works and this really sells and these cards are good they have pie torch support they support multi-card training uh we're going to tile the core and go to a smaller process node cherry 3 is basically 16 cherry twos on one chip you know a big chunk of shared memory between the cores and that's how the cores will communicate no fancy networking they just have a shared address space and a big piece of ddr memory um so yeah we're talking 16 of these mall engines and that would be one pay to flop at one gigahertz uh and then we have a paid-a-flop chip and a pay-to-flop chip where we can actually feed it from the s-ramps uh and there goes to a few reasons as to why that's possible um so our goal with the cherry 3 is to build a 300 watt card that's on par with a dji dgx a100 uh we also will get some okay so you say like how is this more power efficient than nvidia so we are using a slightly smaller process um we won't be thrashing and we have a lot more memory on the chip so we won't be going out to the external memory nearly as much um you know the the control logic is is much smaller uh yeah and hopefully yeah we can make um one pay to flop work this is this is this is pushing the limits now of what's possible with like tops per watt but i think if we have this we'll be able to get all the best people to do it and this will be by far the best chip in the world um is 16 gigabytes not low yeah we can maybe stick more memory on it i mean the nice thing about that is that's like just off the chip ah graph claims 10x energy efficiency over video with the local memory approach yes i mean we're using the same 10x right like it's nvidia's memory that's killing them because nvidia's memory isn't really designed for this um how do we convert tiny grad to binary for cherry you'll see i'll i'll start like you'll write these kernels in risk five and these will be like like like hpc kernels and you just load it into the core um well i don't know what jim's doing attends torrent as for what's tense torrance doing so far ten story's talking about weird researchy stuff they're talking about things like sparsity they're talking about you know this is a bad idea this is a bad idea to talk specifically about like certain kinds of uh you know of models or talking about like the future of what models look like just leave that to the researchers the job of the compute builders is to build very flexible easy to use compute that's available we don't need hbm you don't actually need that much ram going out at that much bandwidth going out to the the uh the things if you have a lot of sram on your chip right so i'm talking about 288 megs of sram on the chip um and this is all for training uh but yeah we can do branches i mean the core is a risk five core so like okay fine the cherry tree will have 16 of them but like still we're on par with like a normal like multi-core computer we're not talking about thousands of cores here we're building a single core risk five super scaler out of order right targeting like i shouldn't even say one plus instructions per cycle i should say we're targeting we've not even put two matrix multiplier engines in this thing um google tpus are just 2x better than nvidia uh well which google tpus google tpus were only two google tpus were 2x better than nvidia um on a 16 nanometer node right so like nvidia's you know down on like a like a seven or eight nanometer node and the google tpu v3 on a 16 nanometer node was 2x better than nvidia um so yeah the the cherry 2 on the 12 nanometer node should be like on par with nvidia um i think my uh my targets for power may be a little uh ambitious i do recognize that also one way to get an easy 10x over nvidia is not to rip everybody off in terms of price all right like we're not looking to rip people off i want to build all this verilog open source here's a vector risk 5 core that i'm planning to work on won't other companies use my ip and what will they use my ip for making ai accelerator chips for the world hmm how many tea bros mit think about like this all right you're only on this planet i think it's becoming really obvious now the way that you get more is not by taking more for yourself if you live in a world that doesn't have technology this is obviously true right the amount of um you know uh uh whatever we're hunting uh deer the amount of deer the tribe kills is fixed so what you want to do is try to make sure that you get more dear with respect to other people in the tribe this isn't how the world works at all anymore everything that's made my life better is not because i have comparatively more money than i used to have it's because the things you can buy are better right no amount we can talk about 30 90s no amount of money in 2000 could buy you with 30 90. now you can buy it for just you know 4 000 on ebay um the world isn't even close to zero sum right so you're like okay so why do you give it away right so what's my goal what's my goal in life right like my goal in life has nothing to do with you know stacking gold bars in a safe or something i mean you know it has to do with uh again i have more you see this red bull right this cost this cost uh i think it was two dollars and fifty cents at seven eleven right um if i was a billionaire i'd drink the same red bull it's interesting like that right it's not like i can buy a 10x better red bull for 10x more money um yeah i hate graph stuff in tensorflow none of it's graph i hate graph stuff i don't want to do any graph stuff i want to build a processor a normal computer processor and i want to execute things sequentially like a normal program no you're not not rich enough to know about the 10x better red bull i know billionaires i've hung out with billionaires i've met them i've seen what their lives are like right like they're drinking the same shit man there's no market above look i go to this coffee shop i pay five dollars for a latte you know there's no there's no fifty 50 latte market oh why is the last step to sell the company um well so you saw the company for two reasons one uh because well you have to pay the investors back um it's not really like comma like solving self-driving cars is very pivotable to other things after you've built the cherry three and you've won the market it doesn't really matter anymore right you can sell it to intel and like even intel would have a hard time screwing this company up i mean they would after like 10 years but for a while they'd be able to launch the cherry two the cherry three the cherry for the cherry five um and so it doesn't matter i mean sell the company right flip it and it's it's not about really like making money for me it's about uh you know alleviating like like giving me capacity to work on other things um so that's why you flip the company right like like in some ways if you don't take their money for it they won't think it's valuable um can you counter node size with density no money is not resources money has very little to do with resources today uh do you see this i mean you can see this in could waymo build open pilot well could cruise build open pile this is actually a better one um right i know less about you know way more sebastian thrown way mostly old days but like um when when you look at crudes there's a kyle vote lex friedman interview and kyle vogt says that his original idea was comma and then he lists three reasons why comma is too hard right um one of them is about needing to support many cars right this is this is the first one he gives he's like well so you know we built crews you know cruise was originally going to be an aftermarket add-on uh we built crews uh but it only supported the audi a4 and it turns out not that many people have that car right well so what open pilot supports like 115 now different cars um the second one that he talked about was uh about the fact that car manufacturers may change the firmware and you'd have to you know deal with that well again we just like looked into that right and like like i think he also talked a bit about like yeah just the difficulty of talking to all the different cars and reverse engineering them it's like yo like we did that and we crowdsourced it um yeah i mean that was and then and then he said something about uh i i'm not even sure i mean maybe he said safety but like again that's not like uh like these are all problems that you can just overcome right as far as we can tell right now from the data open pilot seems now this may just be the open pilot user but they seem to be about 2x safer than the average driver right now you can't really separate open pilot from open pilot users um but you know so like safety car support and reverse engineering done all right sorry you couldn't do it kyle what did you do instead you built the cruise orange right um so but my yeah rainbow's lifestyle commercials it's laughable and that's like like what i made fun of them for um so just to counter your idea that money is resources you know on our slack today someone's like are we sure waymo's not a front for something i mean what what what is are we sure there's three billion dollars not going to like some other operation because it doesn't make sense my plan is to release the chips as pcie cards inter chip bandwidth is overrated assuming you're training these normal size models where you can fit batch size one on a on a card all right no more self-driving cars um yeah so my goal with these things is to yeah pcie cards i mean i say pcie here do i think we're moving closer toward the goal i had in my south by talking about culture and now i don't uh i think 2020 has um i put it on my blog uh i heard a rap song in 2020 thought i'd explain the lyrics here after watching 2020 happen the current group in charge should never be in charge of anything ever again uh so and yeah you know you guys know i'm down to get behind the revolution um i'm not you know i'm i'm uh i'm i'm on the spectrum kid who knows something about computers right where's caesar you know i'm waiting for caesar i'll join his army um where do we manufacture having a single cpu with an additional matrix multiply block sounds so awesome yeah i know right like doesn't it and like like we're we're texting about like i wanted to build these like like like these three registers if you look at my my tiny grad model um like my tiny grad model is uh here this is my if you look an extra cherry it's what i have so this here um like my model i put like four registers and i like labeled them stuff and i'm like yo like what kind of fix function machine crap is this why don't we just have 32 registers and each register is 1024 words i like i think it's beautiful okay so let's see what's in all these vector extensions um but yeah this is this is my rough emulator for the architecture um it supports loads and stores it supports this set of binary operations this is like this is the alu here adds up maldive molac molex is actually going to be taken out it's just going to be a multiply and then an add we got a what do we assist construction set nah brah this is risk you don't have a balac you have maul and ant um and then like yeah we'll we'll take things like that out and we'll give it uh 32 general purpose registers um work from home is an absolute joke uh any company who thinks you can build any reasonable culture with people working from home is so misguided uh cherry computer is going to be in person um either in san diego or la when it's time to hire people i want to get like a beautiful glass office you know what i mean um so no comma doesn't hire remote and cherry will not hire a remote if you're serious about working some working on something show up at the office oh yeah programmers on reddit say otherwise yeah well you know what they also have enough time to post on reddit no work from home's not fine widening fp add and subtract am i working from home right now well yeah there's no office for cherry though comma's all working i went to the office today no no no most of california is not san francisco huh i mean this is interesting right it's like i guess i didn't really realize this but oh so okay here is an example actually of where work from home can work right like work from home can work if you're working on a project that you're self-motivated for right like if you're contributing to an open source project for free you can do that from home you can do contract work from home too what you can't do from home is salaried work and certainly not work when there's equity involved for the ieee floating point standard i think we're gonna not implement rounding uh to match ieee exactly um yeah the vector stops changing that's fine i just kind of want the basic idea i've never really looked at a vector instruction set before my only real experience with it is like intel's avx stuff uh i've never even like coded an arm neon so i don't really know what that looks like i should probably understand it first um this mammal engine would do convolutions versus some stranded offset addressing control yeah um there's tf-32 we're implementing tf-32 uh so on these two we're actually doing tf32 but we took one bit away from the mantissa because all the block rams are 18-bit block rams and not 19-bit block rams and i'm not going to create a whole other block ram just to store one bit so actually it's less of a problem now that i switched to big words so we can reconsider that maybe we will go to 19 bit um yeah we'll reconsider 19-bit all right this is this is the yeah there's a lot of pieces to this project so it's it's good to kind of understand them all so this is in twitch core um this is my memory and like how much striding potential we get is controllable here so like the question is kind of one dimensional strides or two dimensional strides and i've kind of settled on one dimensional strides two dimensional strides require too many addresses to go into the sram um but yeah i mean actually if we just set bits to 19 that should just work let's see i have an 18 here let's make bits an input parameter here every other video i see the same hoes let's get rid of the stripes stride less yeah all right so this is this is like my my emulation of my uh my memory and then we can actually go in here and i should really rename all this to cherry um parameter bits 18 bits bits so now that's with 18 bits let's try with 19 bits oh wait no that's not actually right we have to go to test bench for that what okay i think that's 19 bits hmm that should show at least one more hex character though right oh i said it's okay i said to 19. ah there we go port w well this is expects good way to find bugs reg size equals bits times bets bits times has some size that seems right port five of risk mem expects oh it's this one here that i didn't do right no bits is right there oh did i not pass bits into the memory i didn't pass bits into the memory there we go boom okay now it supports 19 bits now let's see if we can synthesize that and we'll see how many b rams it uses and stuff all right so now it's creating one block ram to store four extra bits but actually that'd be cool if uh if this just worked because it's really less of a problem once we get to yeah look i'm using no did this just work okay seems like adding new devices to pi torch is fairly simple as well yeah it's not that hard oh i didn't change it to 19. okay let's see what changes did i get floaty i finished i did actually yeah floating ad works you can check it out it's in this refill um okay so look it switched all to b ram 18s didn't use any b ram 32s well that's a lot of bram 18s that can't be right this used 65 of the bram 18s never mind never mind 19 bits was a mistake go back boys go back uh did i get my floating point edition to work i did um wherever i thought tf32.c yeah so this is add my normalize is a little stupid but that's my floating point model that's my floating point ad so i understand what's kind of there i also read this good paper about it that shows you like if you use like the dsp how many luts how many registers you need uh yeah wow adding and subtracting need a lot of lots iterative combined for multiplication and addition and subtraction see that looks pretty cool it only uses one dsp uses that many lots um have i tested my core with risk broadcast yeah yeah so i i i went through there's i don't have a like a test harness yet by the way you know i'll make that an issue if somebody wants to file a uh if somebody wants okay so i made the test work but right now you have to go in and like manually edit the test in the test bench it's actually not even in the test bench it's in risk it's in cpu so if you go in cpu like this is the store test right now run the store test i can simulate and if it says trap one it means that the test succeeded and you can see if i like go and i let's let's break the store function right like here we go let's uh do load do store okay we'll break the store function we'll take the data out all right and well okay that doesn't even finish that's trap zero but let's say like we make a typo here and put two instead of one see trap seven that means the seventh test failed one means they all succeeded so if somebody can write something that uh that goes through and test them all i would appreciate you mo asker i appreciate you that's why you're a every vip videos um ooh risk v formal when they do their formal verification cool i'm interested in this too i mean so it really looks like what we're building um i mean tests are good too you want tests you want formal verification really what we're building is a super powerful risk 5 core that's also just a great risk 5 core on its own but also has this insane vector engine attached to it so it's actually i shouldn't talk about like that's not what i mean by work from home is the problem work from home is only the problem if you're getting it's an incentive problem right if you're going to pay people with salary or with equity they can't work from home right like like what they're doing when they choose to work from home is they're prioritizing their lifestyle over your business and like that's great and you know you can scam companies out of that but like that's not you know like we're here to build something right um no it's not even that like if you're contributing to these things from home well you're contributing because you want to and if you're contributing because you want to we don't have this incentive alignment problem and then you can do it from anywhere like yes fuck any biz live your lives what do you want to do you want to you know travel around from from from place to place and you want to like consume things that other people have created the only things that make going to these places good is because there's people who stay there and create value there like something that starting a company has taught me and that i wish you know and and and this is the kind of stuff that i blame on like how hard they make it for small businesses and how how the monopoly stuff um works is that people don't feel like they're an owner of anything and if people don't feel like they're an owner of everything they're anything their attitude is all going to be fucking right like you know they're saying when you rent a car it's like never gentle on a rental bro and like yeah i live that life right like live that life and you know people who don't want to create value like i said yesterday bullets are cheap right so it's fentanyl right yeah you know it's cheap um the employee is not an owner because the ceo or boss can fire them at any time well of course but if you know you have equity in the company you're an owner of the company can you briefly go over why you should not start a business i mean if you're trying to start a business you're already doing something you know who should not start a business anyone's like well i really want to start a business i just need to figur