George Hotz | Programming | Profiling twitchslam | Part5

Key Takeaways

on this beautiful tuesday morning i put my phone into charge where's your phone bros bought an apple watch so useless explain to me what i would really do with this thing i really don't know it's dead too even better well i had a little display of wearables that i bought that nobody would ever use good morning campfick all right guys so let me tell you why i haven't been on twitch for a bit because we've been out of mate but today we have green mate yerba mate enlightened mint this is the best one though i used to think that i'm having it now i took two sips it's kind of mediocre you know what i want twitchies waffles get me waffles um no i don't know about any waffles but all right so let's uh let's see what music we're listening to today and then we'll get to it and we'll get to working oh yeah i'll be listening to this song you know when every other person here wants every other to leave that's right san francisco is truly a tribal city you guys it's home to different factions of people who all want different things and all dislike the other factions we have our very own set of morlocks you know the morlocks who read the time traveler ty the time machine how was that that was cool ah the more locks all right so let's go um what do we want to do well let's focus on let's turn the optimizer off let the optimizer off but we'll just uh we'll fix all the vertices and then let's focus on removing the low quality points look at all these low quality points those shouldn't exist eventually go away i think i think my pruner eventually works but like it'll work a lot better if we could just not add them all that crap all right so let's read let's read orb slams filtering we do we do like this filtering here but let's see what orb slime has to say i know warp slim just has like really fancy ass shape for this here we go triangulate each map what's up new gum cruise i'm actually hanging out with that girl tonight um you know we still hang out okay check parallax between rays let's figure out what that means let me see it works on paper do you guys know what parallax between rays means search for matches that fulfill epipolar constraints so that search for triangulation so that search by projection i think well this isn't search my projection is points that are already in the map so this is about adding new points triangles find the point 25 new map point creation [Music] connected keyframes so each unmapped new map points are created by triangulating orb from connected keyframes in the co-visibility graph for each unmatched orbit ki we search a match with the other unmatched point in the other keyframe and describe those measures that do not fulfill the epipolar constraints orb pairs are triangulated to accept the new points positive depth in both cameras parallax reprojection error and scale consistency are checked initially a mapping is observed from two keyframes so that it could be matched in others okay so yeah guys i like microsoft i don't have a problem with microsoft they've never done anything to like really piss me off um i tell you i probably like them better than like google um i don't know i mean i don't really just like google either but like fundamentally microsoft's business model is much more aligned with my goals than like google's business model um like it's unfortunate that like google and facebook are media companies talk about this a little on twitter uh it's unfair it's unfortunate that they're media companies right because the media company does not have the same goals as you whereas a company like microsoft or apple i mean they kind of just want to like you know they won't like sell your product right like they want to sell you a product or service that's fine that's like the engine of capitalism that's what makes capitalism great um like really i mean capitalism when it's based around consent um i mean that is consent right when i go when i purchase an iphone x this is a consensual transaction between me and apple when when google you know tracks my cookie around the internet and then like uses psychology to figure out what's most likely to get me to click on an ad it's not so consensual anymore right no just wanting money is great guys don't don't don't get it twisted like capitalism and wanting money is great um the problem is like you know like it's just i mean you know you could feel oh god we won't go there we won't go that's too much but like really what it comes down to is consent okay um ah yes model initial pose estimation new map point creation okay we check for positive depth in both cameras parallax reprojection error now if i recall correctly orb slam just does the triangulation in yeah it's just doing it in the real world it's just doing it in like full on uh current keyframe so we we had this for a bit but i didn't think it worked all right this trip is just doing it in the actual pose of the frame but when i tried this it didn't work it had like numerical stability issues um what do we know capitalism is always consent it's like advertising that can kind of blur the line right i mean what is advertising but kind of a form of coercion um you know you don't want coercion don't content oh okay i think i actually know the reason it broke ah this is interesting okay i think that it's just because my 0.1 check was bad so let's let's go back to doing this and let me like the checks just in line here did i break the google chrome sandbox i don't talk about that um okay does that triangulate the points we don't have matches for so for some reason that i'm still triangulating them all we don't really have to do that well do we no we don't okay so they just say equals equals zero which is ballsy sorry i'm waiting for uh i'll try texting you this afternoon i'm both until okay i'm talking to a reporter later today you know we gotta i gotta say nice things to the reporter no nobody calls an assembly language don't be don't be a noob noobs a lot of noobs all right um let me just say not equal to zero let me try that we certainly don't want to have divided by zero that's good i'm not even really sure what that does um okay now we're separating triangulation from the checks okay now let's do checks down here so we check that it triangulates in front of the camera so basically what we want to do i believe like you can really see this well on the kitty reverse one uh i wanna and i probably should have gone to the hospital but i decided i'm going to refactor this a little bit is [Music] wow [Music] so if you dot it with the inverse of the pose you project it from the local frame into the global frame which makes me think that if you dot it with the pose itself like is that the same thing like i don't know why they god look at all this complex shit they write why is there all this complex shit um [Music] is that so crazy that that that i don't like why is that that's just wrong i'm sick of math bros okay so it's not like right rcw row two dot oh so they're really only computing the z let's just say compute couldn't care less should i do this is this more right it seems more right okay so we have to check that it's in front of both cameras so you can see like this point here is behind the camera as well so let's just think this through and think whether i'm doing this right so we're moving forward so we're moving forward on the z-axis so we would expect that the points when moved into local camera frame are if we're at six and this is zero no hacking anyone talks about hacking's banned everything's illegal bros don't hack uh all right so this is a zero and this is six so we would expect that if the point is at 66 in global coordinate frame it's not like 60 and low corner frame so this is correct right um so i don't know why but that's correct okay well so this is a orbs there's a cv2 camera thing that i really like this just shows you a good way to think about it this yeah so that's our pose matrix and we're dotting our pose matrix with the uh okay so you see look we decipose matrix and we dot it with that um so we end up with so we'll say you know p l l1 dl2 if pl1 sub 2 is less than 0 or pl2 sub 2 is less than 0 check points are in front of both cameras i don't like this equal zero here when do things equal zero i'm not sure what that means in a floating point world this is not literally identical it equals zero okay it was like somewhat less crap okay check reprojection error in first keyframe check reprojection error and second keyframe okay i don't know why they wrote this like ridiculously complex shit um i should just be able to take that point i don't understand i can just take that point and i can go like this right empty dot k okay here probably move this out of the main class bro get out of hacking hacks are losers there's cool brows math do math that's cool very cool so those are the x and y points now we can actually test the reprojection error i mean pretty cool so those are the errors in the reprojection if we assume that that's the point so we can check them they all look reasonable oh i don't know what an allowed arrow should be greater than one or something five nine nine one oh so they're not using mean squared they're just using i just mean squared they're just not taking the square root [Music] um that's cuter 5.991 times sigma square 1. okay i don't know look at this one seven greater than one [Music] what i say about hacking equals bam wants to be banned okay i think the last check i need to write is the parallax check it would be helpful if i understood what parallax was so bitcoin subtle there's always some sucker that will buy and machine learning is not limited by data it's limited by infrastructure okay let's let's check parallax check reprojection error check scale consistency i don't know what that means i don't think we have that oh you wouldn't we don't have scale let's talk about that mlml's kind of cool it's not self-driving cars we're really here to talk about slam though um okay check parallax between rays let's understand this in fx i don't understand what this stuff is oh okay so this is just dotting it with kn which i think i have already so we can take this i'm going to check that we'll check that up top um that's in helpers uh we should just check this for all of them cool do this kind of shit somewhere else too uh i don't know if that transpose is really okay it probably is yeah it's like a fairly symmetrical matrix okay so that's just projecting them into normal frame so let me see what this parallax check actually means check parallax between race okay well so that's xn1 they call it xn1 and xn2 copy their naming convention i don't really know what that means but rwc1 so it's the rotation of the keyframe i don't really understand this no stereo and very low parallax well first off that's a bug guys you should be checking for my bugs okay so we want to this is a rotation based on the pose so the copy there is exactly and not really understand what it does it's this ray one dot ray two i don't i don't understand this why are they rotating oh kind of get it parallax is large now we can use okay so right one dot right two so i mean i guess i just call this array one i don't exactly understand what array means parallax is large enough so oh they're doing xn1 there oh that's kind of cute um i do that with triangular right no i don't what how does this work oh cause i just don't do it with kpus okay this is this is stupid we don't actually need to dot these k ends because we actually track that already so we can just do kps here by rotating into pose frame i might have that backwards i have to do sub idx right yeah could you update your readme section in libs cv2 misleads why i don't think it matters so called cv2 i don't know if it's opencv it's not open cv3 or iphone import cv2 um typography it's python 3. still important cv2 right okay so they're dotting them together and dividing it by the norm so this is apparently whatever parallax is it's always one like this might only be like not close to one in the presence of crazy rotation or something this does seem like the most useless metric or i did the math wrong okay it's like always one cosine parallax rays is less than oh so we don't even have a cosine power exterior because there is no stereo the fuck is this shit b stereo one b stereo 2 and cosine power x raised is less than 0.9998 if parallax is greater than or equal to 0.99998 continue let's see what we get loss of a lot of points but is the crap gone that's what matters no we still crap we should have more crap did i get the check backwards no i don't know what this means we want to look for high parallax points i have a definition somewhere if uh scene is planar or newly planar and there's low parallax can be explained with a homography parallax is checked okay so basically yeah points at infinity do not have um sufficient parallax now sometimes not including them on the map and are very informative rotation of the camera you know shit okay cool um huh they're parallax now so they did definitely the other one the other check i understand like most of the time this thing doesn't have stereo cos parallax stereo equals cos parallax rays plus one and b stereo zero b stereo two or more so that's the only check that matters um i don't know like it just looks like it throws a lot of points away let's try it out the ground truth and see if that works are we we're just throwing a lot of points away when we do this oh that looks better oh that looks significantly better take a look look at look at how it's not like you know that's so much smoother wait except for the fact that it's so smooth that it's just flying off the end okay so actually it becomes under bullshit because it's totally wrong so this check is garbage wait zero vertices to optimize oh because i'm not finding the frames okay different problem bros put that back to 20. otherwise it gets really slow okay so now we have we have jerkiness this is without the parallax check this is vaguely correct let's add the power last check back in i don't know it looks kind of the same it's the same kind of low quality shit look at that low quality shit i think the map might be higher quality and i think we might be getting less crap but are we just getting less crap because like we're adding less things oh i love money guys i love money money is great money is you should want money money is what makes the world go round love money love capitalism money some people don't have money yeah yeah that's a tragedy man but i got money i love money you should love money too that's right let's see let's look at some money oh yeah look at this twenty dollars yeah bros look at that money ha ha ha i love money monitor's dirty don't talk about hacking we're not talking about hacking bros i'm talking about slam that's right and money they're talking about money that's right twenty dollars yeah get it look at all this crap being added to my mouth look at all that crap i thought that went over there oh i hate this check i don't understand what rays are man that's array the other checks are all very reasonable noise so i use two for my reprojection army we should up this to two let's be a little more reasonable here how can you have money i don't know earn it perform a valuable service for your fellow member of society it's not open tv3 this is not it's wrong wrong like not having money have money the problematic nature of this twitch no no no no those belong there no that's never going to match up those points ah my map is ruined you want to earn money from computer you can earn money from computer there's so many opportunity to earn money look at that shit look at it doesn't really do that when i do this one it doesn't do that what would i do because it's not up because it's optimizing but like um update normal and depth we should probably track normals as well it might be a cool thing to do what do you think about working no no no no no no no no that might give you money but what you really want is passive income just imagine the problem with working for google or a big company to have money is you have to work imagine you found a way where you could just get money you know what i'm saying like you don't have to do anything you just get money what a deal that would be all right let's let's change this a little bit it's well uh i don't know about this yeah bro work is life but you think you want to do your work for somebody else man you got to get equity you got to work for yourself you spend your life making the salary let me tell you something you're never gonna be rich you're always gonna be working you gotta you gotta get some money you know maybe maybe put that shit in the stock market bro figure out how to invest and don't invest in dumb shit invest in index funds that's right if you invest in index funds invest in you know what pewdiepie says invest in property bros um if when x dot shape equals equals one then we just want to add a one to the end oh come on should i just write this out right code terrible that's that's still there no i could just say learn what parallax means i don't want more viewers i like having a few of you bros if i have too many viewers it would turn into lowest common denominator trash like our education system you know sucks index fund's great oh you don't have money to start the work well that sounds like a bootstrapping problem i don't know how to fix that that's you have to be born into money you don't be born into money that's fucking bullshit man i don't know where you're from but this is america and america is greatest land it's a land of opportunity and there's so much opportunity you can like pick up a rock in america and there's opportunity underneath it probably i don't know man dig into time series model i don't know yet the trick to getting rich is okay step one find somebody who has money step two convince them to give you that money but don't do this in a scummy way don't don't like be a creep about it you know what i mean i mean you could probably talk about the same two steps to getting laid but you know like don't be creep about it man like don't be like sketchy about like how you make your money make your money like an honest way you know oh yo yo i do love taylor swift that's right that's right i do love taylor swift uh i don't know though i don't think i'd actually really get along with taylor swift um yeah we should commit this is pretty good this morning all right we have a local window again fixed up that we use add ones we do the triangulation global frame we have a commented out check for parallax cool change what's my kind with the lips of selena gomez i don't know that much about selena gomez guys um i just know that that come and get it not on our song was about justin bieber being afraid to lose his virginity um what mac do i have like what's like the most expensive macbook pro you can buy at the store not like special order because that's too much but like go to the store like give me the highest then macro pro that you guys sell um because i'm rich and you should aspire to be rich you should aspire to make money because let me tell you something if you're making money you're contributing in a positive way to society and that's what it's all about guys that's what it's all about doing services and goods for your fellow man doing services and goods for your fellow man how much do i like trump we're not gonna talk about trump guys no politics stay out of politics bros politics is the mind killer you know let me tell you something like every time people read these like political articles and like oh my god this is happening or this is happening i'm like i don't give a shit man though i am excited about uh about peace with north korea because we love peace peace is great um thank you be friends i'd like to go see north korea like i kind of want to be one of those like i don't know like i have a friend who went um i kind of like to be one of those people that goes like maybe right when it like collapses yeah it's a bill gates i mean yeah this is right you know it's a god you know what i didn't like that bill gates said though taxing robots that's dumb as fuck um bill gates calls for robot tax no no no no that is so dumb um i don't like taxes but not a political way you know like i'm not gonna like campaign about it i'm just gonna like mildly complain while i pay my taxes because that's what you do in america you know they say that there's two things in life you can't avoid taxes and death and i'm okay with paying taxes i would like to avoid the uh death though i think maybe maybe if we all work hard you know we could avoid that good so this is good that this works um because it didn't work before when i did it in a global frame but it was because of the what do i police in new jersey they attacked a woman i don't know man that sounds like politics um i don't know you have to also ask the question of where did you hear that from did you see it in person like if you saw it in person then yeah i trust you feeling pretty fucked up but like time series model is two steps to get laid no no no getting laid is like getting money convince someone you know you you find someone who has that and then you you you know you you uh convince them in a positive way no coercion to uh you know to to to share it with you right now it's the same thing with money it's all the same shit bros what do you think about break bitcoin code and bear is that a positive impact on society that's the question you have to ask yourself it's not just like if you just like be rich and you're a dick don't do that be rich and have a positive impact on society all right so look at how good that looks until like it doesn't look good anymore see when you do it from this angle it doesn't look as good what is all this shit that shows up all this like speckle does the rotation one work i feel like that one doesn't work at all worked a little bit if i had to parallax check it it'll work oh it's kind of better with the parallax check well okay so part of the problem too is we got rid of the kinematic model we have this or true in here but then we just get this kind of behavior which this actually is not the ground truth for the thing as far as i can tell yeah like looking i'll just keep changing that way so the kinematic model is crap oh no no it's always other individuals there's only individuals bros all we have is individuals in the world everybody let me tell you something you're born an individual you'll die an individual and whenever people try to talk about how it's for the common good like oh oh do it for the the common good i mean that's just like that's just their way of saying me but they're trying to be more like politically correct about it right like like you should you should fund this project for the people and then like you look at the person who's saying that you're like but isn't that like your project aren't you gonna like make a lot of money no no no it's for the people don't do that don't be one of those people be like yes it's for me and i will get rich but you will benefit as well and then everybody wins right like don't be one of those like you know what i'm saying um do we not so we can talk about elon musk you know musk is a bro man we love elon musk elon musk is doing so much good for society elo musk is going to get me into space man now let me tell you about space look i love space but like i don't want to go to space until it's nice but what's great is there's all these people in the world who want to go to space even when it's not nice and they're going to make space nice and then i'll go to space when it's nice um am i a better program with the law mosque yeah probably now okay so the optimizer just messes everything up there default 11. great why does it do that space have different gravity i've heard this yeah better watch out for space gravity why is this seg faulting what seg faults i don't understand are there nands going into the optimizer oh they're nams going into the optimizer no nans so that's egg faulting i don't understand okay okay i know i know what quit unexpectedly i know i know you're sad about that i don't even have gdb on this computer i only have ll db make space nicer again that's what i'm talking about man i want to go to like man the intercontinental space ritz carlton space mandarin oriental space i want to go there right mandarin oriental space but don't be poor because if you're poor they won't let you go to the man or an oriental space got to be rich you're this point look it's so far away look how far away that point is it's so far away that points practically look at this point this point's practically in space i think about quantum i think it's full of scammers man i don't know i think that p equals b p equals b q p no i'm sure someone's going to link me to the blog post say showing me that isn't right oh i know why it's segfaulting oh that's a stupid reason it's like faults when it can't move any frames for some reason like the problem is when you have almost purely rotational movement of a camera you don't really get uh you don't get depth information of the points so you can't make a good map honey rise of mars that's what i'm talking about they gotta make a rice shop on mars first though because i don't want to grow my own rice that sounds hard what's my comment about nasa versus spacex what do you mean why is it versus bro anyone who wants to go to space is cool nasa can go to space spacex can go to space you know what's really cool about space is really big there's plenty of room and space for everybody and maybe we don't have to like be dicks and like fight over space maybe we could just be like you could have this space and i could have this space and damn there's so much space i would love to see humanity colonize all of space the match is just all of space imagine there's like space mcdonald's man when are they gonna build the first mcdonald's in another galaxy when i get many simpler videos that serve as a test case instead of just watching this one all the time yeah i mean that's kind of sensible i should probably do that i should probably get more test cases well so a few things i want to do use the alt optimizer is the alt optimizer better it's not better it's the same shit we could get many simpler videos i mean that's okay so that's one direction we go in this is we talk guys i didn't smoke today uh is there some way you could get a dot product to map i don't know what that is so part of the problem with other videos well there's other freiburg videos let's download another freiburg video um this one looks cool wow look at this efficient online surface correction for large wow that looks pretty nice so they're using depth info though okay so those are for testing and debugging um floor desk desk 2. room room that sounds good let's get room room why call she he no no no no no no no no there's only gonna be beautiful women on mars that's what they promised me man have you ever read any bee comic about space that's right no i don't know about that man i think there's better things to do on mars to be honest like eat rice all right this is downloading you guys can't see it download because i'm covering it what's fr1 and fr2 i don't even know oh this is a rolling shutter camera what a scam this is rolling shutter camera let's see what like uh lsd slam data set you know what i can actually they talk about this in the orbs paper don't they like what data sets they tested on all right so this is kitty they tested a lot on kitty which kitty is this because some of the kitties are better than other kitties they're also doing loop closure which i'm not writing loop closure they didn't use room i didn't see them use room look at that accuracy though they use desk um all right you know what let's just write a proper what now i did this in one of my previous uh here we go oh man you know we should track i think my apm is kind of low man i should get my apm up you know copy and paste this and just we'll stick it in the parse ground ground truth timings dot txt yeah cause that's rgb.txt this index out of range i need to save it as tip now we have room let's see what room looks like room our room looks nice except why is my slam so crappy i think it would make like a nice map looks like shit i'm getting troll mod failures uh why is my slam so crappy these points are crappy okay that's arguably less crappy wow but it's still not it's not crispy okay so one of the things that orb slam is doing did i do the mouse on tmux i don't think i used the mouse ew that's gross one of the things orb slam does that i'm not doing is orbstein uses keyframes i am putting every frame into the optimizer which i mean you think like more information is better right but that might not necessarily be true that's a pretty nice map and a bro stack is it a nice map let's get rid of the ground trees let's see how crappy it is now so we're not using that constant velocity model we're using crappy see now it's like messed up oh no we got an infinite jack hate that infinite jack yes so the problem is right now that it always assumes motion because that's how the i mean what we could do is we could re-scale it so let's say yeah hang on initial positions for fundamental matrix if frame id greater than 5 what we can do is we can get the velocity here and we can say velocity use kinematic model for velocity um well so velocity right now is like a crazy like pose-like thing but that's that's wrong um that is now going to print out yeah so you see that's like a velocity so we can actually scale it based on that um velocity right and now we can say f1 that pose divided by equals magnitude no wait do we want to divide the magnitude or multiply by minus two that's a little better right now it stops all right good it doesn't bug out as much now up no okay well it just goes crazy now because magnitude goes crazy but it shouldn't damage it feeds back on itself no okay this was the crappy idea don't do this stupid or slim actually just uses a kinematic model like this and it works when i use a kinematic model i end up with this oh troll mods yeah so if it loses translation it doesn't recover translation this is a problem in orb slam as well money is kind of dirty that's true i wouldn't like a money the problem with the fundamental matrix though is like it doesn't scale the if you don't have a translation component that's very large it doesn't scale it well [Music] well it survived longer now look at that open cv book oh that looks nice it's a crispy part of the map and how it jumped to over there that's not what it's doing at all okay now it's like broken why is it so crappy this part of the map's kind of nice stop being crappy okay what dude you almost be famous fuck that just go do something gotta get that discipline that's true you do need discipline in life okay well that didn't work god why is worm slam so much better actually i've never actually seen orbs lame run on room so maybe it doesn't work this is with ground truth and ground tree helps it it doesn't troll mod like crazy anymore those are all wrong oh no it's not light so much peace like my mouth doesn't look crispy you know what else well no i can't really do that we do run the pose optimizer at every frame right oh no we don't because we have ground for posts so we're just sticking it it keeps jumping over there that's where the real poses fix yourself slam something look at that bro look at that bro yeah stand still bro if you're a moving object you're bad look at that bro stand still he's famous he's in freiburg oh my god we literally pass right by a white wall that's unusable i guess that's why nobody succeeds at room nobody uses room download one more than web slam uses desk and floor let's download desk desk so there's also depth information we could potentially use but that's kind of cheating and notice how they don't use so notice how they only use xyz they don't use rotation because i don't think it can do it we know web slam succeeds at floor and desk and xyz um okay desk see how desk looks this ground truth okay like but even look like this is all messed up like this is the same this is ghosting it's the same thing twice right those points are all garbage we know orb slam does almost perfectly on desk the real problem is because my crappy uh do we need key frames is that the problem once them talks briefly about why they use key frames bundle adjustments i don't care about real time uh so i almost initially everything was processed shortlisted in the math location yeah okay so this the accumulation of linearization errors i don't really understand what that means um because that would involve like math but like i've heard the same crap with kalman filters too um all right i mean this is a great video to test on like you can really test almost everything on this video um but let's try let's try this let me try one fourth of the frames no so that's just doing it with less frames it actually seems to kind of work better look at the map is crispier i don't know well that just messed up but like you can't see what oh you can't say oh you can't read the text oh sorry yeah okay i've got today's work on slam tomorrow to do um in yes okay if we need keyframes tomorrow but yeah so that's what we will do tomorrow the idea of key frames is instead of putting every frame in the optimizer you only put some in the optimizer there's linearization errors i think is what's happening now and i think that's why webb slams a lot better than us still um so we will uh look into that uh tomorrow i'll see you twitches tomorrow it is nine in the morning welcome welcome good morning vietnam uh we're resting peace robin williams uh that's it all right see y'all later