George Hotz | Programming | so how does twitter work? | API | GraphQL | requests | Backend | Scala

Key Takeaways

it might be live all right so now I tweet this foreign you know I figured it was most appropriate to live stream here on Twitter instead of on uh on on Twitch since we're gonna figure out how Twitter works on Twitter should we should close the closet door um for those of y'all who don't follow me on Twitch I'm a streamer uh Twitter's not Instagram stuff's not copyrighted um if you want to post it it's entirely you can no clips obviously you know my rules uh but let's go okay so um the way that I got this by the way uh my friend who is watching my Twitter text me he's like yo I can get you media studio um uh yeah so you know we got media studio uh not through any uh legitimate channels um so Elon you know this is free money I would have happily clicked Bing uh dude all right uh so we can uh I'm in OBS here so I can go like this and uh we should be good that should be my screen all right cool uh we're gonna move OBS over here and then we have a nice Chrome with uh clear history here uh all right and bring that up all right so these are my Twitter notes so far uh you know you can all probably find this funny if you work at Twitter um so we're gonna do this without being logged in I just go to twitter.com uh you see what comes up here and then a great place to start with any web stuff is to start looking at the uh is the resolution good enough for you guys to see this these stuff I know people complain oh well it looks like it's being cut no it's not being cut off my mind oh okay that's kind of visible um there's also a lot of lag there's also a lot of lag uh we don't have any live chat here so you guys can post on this tweet and I'll read it over here um if you want me to uh see something oh my God okay so I was already looking into it a bit um it looks like so a few notes about Twitter it doesn't load at all if we turn JavaScript off uh so if I just pin no script here uh if I turn no script on you'll see that it just complains it says javascript's unavailable so that's the first thing you do because if you can do things without JavaScript it's easier but the first thing Twitter does is loads a whole bunch of JavaScript um so we can look and view the page Source here where's my JavaScript um oh there's a no script tag that's some that's some OG uh yeah so it loads these things I feel like that number actually changed is that like a cash Buster uh copy and pasted which ones actually was but yeah um so I can go here and I can I already have cache disabled uh so we can go here and see the JavaScript that it loads because it's not being loaded it was being loaded last time I looked though so I don't know what happened um regardless it loads JavaScript from uh oh well so I can change my notes up a little here this is what webpack is I I don't look I'm not that familiar with front-end stuff so you know if you're expecting for me to be an absolute front end wizard I'm not uh you can all watch me uh you know be a noob um but we know some Basics we know the old school internet uh yeah so I I noted a few things they have a CDN at twimage.com if you go to twimage nothing loads but if you oh sorry um if you do a who is we can figure out when they registered it it's old registered so we're docsing Twitter I hope this is okay we're doxing Twitter on Twitter it's like meta I don't know um how laggy is this live stream by the way uh let's let's refresh and see what people have to say in the chat let's go not that visible is this recorded low res let me see if I can do something about that um custom here put video bitrate where's resolution can I can I stream in higher resolution we have a lot of bit right here uh we used a custom I use the custom and I copy and pasted the r I don't want to show you guys any stream keys or anything no we're streaming and oh I guess we're streaming at 1080 and my streams my screen's a lot more than 1080 but you should still be able to see um hard to read any text all right we'll have to just zoom in a bit more all right well we'll zoom in we'll try it zoomed in you know it's harder for me to work zoomed in but you know what we're here to have fun anyway um okay uh so yeah this is what I found kind of from the front end um now I don't know if this is the same API that uh Twitter does but there's a really cool thing you can do so let's like go to a tweet here oh let's click these does this truck make my dick look small great great God love it um so here this is tweet detail excuse requested URL there's actually something really awesome in Chrome you can right click and go to copy and copy all as curl and wow I'll copy all oh that's not what I do okay okay yeah great great great great great great yeah yeah yeah yeah okay we want to copy as curl not all of us Carl uh so this is this is a curl request and we can break down what these things actually a minute oh let's just kill this there we go um just create a new file here called curl.sh and actually let's start making some requests from python y'all want to see me do some coding you know we're here to code am I violating the Twitter terms of service probably but I'm doing it live on Twitter so maybe it's okay um okay okay that crowd works all right so let's break down what these requests actually are um so for those of you that don't know graphql is a query language for your API and I think there is some ways to request like some metadata now I don't know if this is the Twitter developer API so let's let's always read the docs if the docs exists first uh okay it's readable yeah what are you talking about a resolution and proof how do you go live on Twitter bro you gotta have a guy you gotta have a guy get you access to Twitter media Studio man like legit I don't know if there's another way uh but yo my friend texted me this is this guy's like an internet like he's an internet genius you know with all like the people he knows on the internet uh and he's like yo I can get that for you um so uh yeah let's see if the docks here uh say how to do it uh what was I looking for okay let's let's just search for graphql and see if it's the same API your search for graphql returned zero results okay let's go to uh realapi dot Pi okay we're gonna call it real api.pi because it's python um let's first find a request that returns a less expensive amount of data I recall correctly this one does bad guess token oh well oh well that's a bad guess token okay um let's find a shorter one of these okay so this is a totally different API this is API 2 guide Json I also found this thing called hashflags.json um it's kind of cool it like shows you like probably the people who paid Twitter money to be a part of this thing uh yo this is so fun boys it was great you know what's great about not having a real job at a place you can like just you know mess around publicly uh and you know it's fun it's just fun man because there's no expectations you know once there's expectations uh so I know that we're on Twitter now we're not on Twitch so some of y'all might be new to my stream and new to the kind of things I do uh we kind of shit talk casually while programming so let's look at this here copy copy is curl all right that one runs and emits a less offensive amount of data so let's go here um so let's throw some V's on that bitch we got vs uh good wow API V2 HTTP V2 yeah Twitter uses HTTP V2 okay so fundamentally first off interesting thing to note is that it's a get request so let's start with request.get and let's copy and paste this uh yo I was I was listening to you know some gang gang I listen to gang gang why how do I do this the quest dot run it feels like a way to do this ah see I don't know anything don't hire me I don't know anything like you think George you should know how to use requests man if you are a real senior software engineer you'd know that kind of stuff but you know I'll ask I'm a fraud um okay so I will note that that actually does nothing uh let's look at the status quo too because I'm probably getting a 403 I guess it's gonna be a four or three all right so for those that don't know 403 means unauthorized you just got to know your HTTP codes that's a question I throw out in common interviews I'm like you know what's the 200 all right what's the difference between a 301 and a 302 man you know what do you think of the band 303 now I've never asked that um can't trust the hoe we won't trust as a hoes don't trust me uh we're gonna need some of these parameters because they're important oh we should bring the mic up here too we'll give you oh yo I tried to get better lighting I almost bought expensive cameras but then I bought a ring light uh so that's why my face looks like I'm glowing ah oh wait are we allowed to listen to copyrighted music because this isn't twitch should we just should we do it I don't know man this is weirder talking to myself without a comment or stream being like George no no they'll mute you they'll kick you off the internet if you listen to copyright music while you stream um all right okay so let's figure out you know what first let's you know what I always say hacking is just binary search by hand so let's figure out which one of these actually matters so notice that the request still works I don't know let's delete these ones and see if the request still works all right you see what we're doing we're doing binary search by hand good the request still works great uh do we need those ones what is dash dash compress do for current nope we need some of those some of those are important compressed oh and automatically decompress the content so I don't think we need that uh I probably should have done this in the bash script so I'm not copying and pasting things over and over again all right good that still works which of these can't be important uh Authority cash control not a brand that's those don't look important still works so we're just trying to figure out which of these HTTP headers we actually need to make Twitter requests um probably not important but maybe important still works X Twitter active user all right we're doing web boys we're doing web who likes when I do web right we need this serious health token um I didn't even need the csrf token okay great um I guess I just needed the cookie or do I need the guest token let's try without the cookie the cookie was not important maybe just the guest token was important I don't think I need a user agent internet see look look at all the stuff I've already removed we can remove all that stuff and make Twitter that much faster because that's how it works okay we need an authorization and do we need a guest token let's just do that last thing see look look look look at how much I've already optimized we were making all those requests with all that stupid stuff and now we don't need all that stuff okay um programs data headers yeah that's it let's say headers equals headers and let's throw some headers here so I mean there's a lot of ways that we can go about approaching something like Twitter so this is let's figure out just how their API works and think about what we might be hitting I started some notes on the back end usually I mean you can't get much information about their back end but usually when you want to look into something like this you read their engineering blog and it's usually pretty good they talk a lot about Hadoop uh I will also I will be honest I've never written a line of Scala in my life so maybe we should just take a quick look at what Scala actually looks like um learn Scala should we learn Scala uh Gauntlet installer I you know what maybe we should do some hacker rank programs in Scala all right we'll get there we'll get there come on I got to prepare who knows who knows what's gonna happen at my Twitter interview guys you know who knows um how do you go live on Twitter you got to be a giga Chad to go live on Twitter bro uh uh can't really see anything quick grab hotz's bear token and go Rogue what's my Bearer token did I post up oh oh that token guys I'm not logged in because you can't you can't do anything with this token I'm not logged in um that's why I'm not logged in guys uh do you think I'm stupid do you think I would do this with a logged in account bro I have some off stack no I'm I'm okay I'll check no one tests my op sack I'll fuck you up in court bro uh headers we have X guest token you know what this reminds me of I did spend some time automating uh dating sites like automating Tinder and Bumble and stuff reminds me kind of doing that let's throw a shebang shebang by the way who's excited about the new python python is it python 312 um okay good so we'll notice that I've replicated the curl in Python uh okay now let's break this out a little bit um oh God look at that uh uh I want to decode those things it's not like a way to do that like not gonna first say URL I URL all right there's got to be a way to decode this I'm sure if it's gonna work um python decode percent encoding parse import unquote whoa okay so basically that's just Json and the variables can actually what if I just do this can I just still pass that to requests hello we're streaming on Twitter no it wasn't but um our boy we just say we're not going to say on stream but the hook this up and got us Twitter media studio so we're streaming on Twitter now we have 857 viewers yeah we're reverse engineering Twitter on Twitter yeah so how does Twitter work right okay so good good I'm glad you know they have safety mode user Fields guys safety is very important um yeah yeah yeah yeah this is this is Alex you all know Alex do you want to say anything to the stream he's like I have a light now it's in the wrong place I do the light wrong do I have that light over there look how much oh I know okay we'll lower the light you want to you want to be in charge of Lights I bought a ring light guys you know that you need to be it should go back there yeah I know but that doesn't fit back there we'll just lower it a little bit yeah yeah is that better yeah yeah all right I think even if I just put it on the table it'll be better no no but then we're not messing with the light too much okay thank you I'll be done with this in seven hours Alex I gotta prepare you know if I don't get this job how are we gonna put food on the table what yeah I don't know what they're gonna ask me maybe I should learn Scala [Music] now listen do some scholar challenges guys we'll do some style challenges oh that sounds lovely um let's say how do I describe percent encoding like right uh let's just take this and throw this in here okay hang on I want to do a few things um let's call it uh what's called variables it's called Data variable somebody's sketchy data and then yeah you see that I have to like Google things too because because I'm actually an idiot um all right so we have params equals variables let's see does that still work oh the following features cannot be null look at this we're getting some data back though uh headers params variables data shouldn't that work um is that what params are dictionary list of tools I mean try that does that work hmm weird why does that not work isn't that exactly what params is list of Tuple or bytes send in the query string for the request okay I don't really understand but whatever um wrong I don't know regardless uh let's break this Json down so uh Json Json dot dumps yeah good you understand sweet all right um let's see if this is normal Json all right oh no loads please get that backwards extra data okay is this not valid Json um why is there extra data data line one column 89. it's like a broken quote or something oh and features oh I see okay so that's why that didn't work um because we have something called features too okay so first we have variables and that's equal to this a user ID now remember you know you can say like George why you wasting time copying and pasting and the truth is this is how you learn at least this is how I learn and like I said you know I'm slow so uh the truth about being a 100x engineer is just putting in 100x more hours no it's not really true so now let's see if we can get params to work [Music] variables variables I promise we'll clean this up but this is using the undocumented Twitter API or it's probably actually documented somewhere and if someone wants to post like a link to it I would appreciate that but we're learning okay that doesn't work okay does this still work oh great I'm glad that works otherwise I would have been annoyed okay okay um can we Json dump these jump jump oh let's just throw a gear repo here just so I can whack just so I can get diff when I break things I'm probably not going to push this but uh it's so I can get deaf when I break things okay again it's loads well those automatically be Json encoded well they won't be uh wants to write a dictionary comprehension params equals okay Json dot dumps X for X comma K in params.item see I can do some stuff without having to Google probably got it wrong I got it right all right cool uh I agree so we have headers we have variables and we have features to start looking at what our response is okay so our response clearly looks like Json so I think there's actually I can I do like r dot Json yeah so on method response um sir r dot status code equals 200 200 is good guys when you get a 200 you're winning oh good good they have a Ukraine flag and they have atheist and resist and Panasonic is that good that's good let's get to another one of the good people um it's not use this this is junk uh variables sorry I just picked a user at random I don't mean to call you out but I mean I guess I'm shelling for your causes so um well we're going to try incrementing the user number by one now remember when we've incremented the user number by one he went to jail and became a Nazi so let's hope that doesn't happen to us when we do it [Music] let's try eight do we get a new user I think we got a new user guys I think I don't know I can't really tell uh there's a python thing to like pretty print Json python pretty print Json oh actually there's just a pretty print in Python pretty uh okay so that dumps no no preprint all right look at all this great stuff we got um and now this should work for any user ID all right should we should we sponsors webql timeline enabled all right so we're making a query to Twitter that has this stuff well probably like do you like George you can just do this all in this simple web tool and yeah yeah you probably can but remember I'm slow so uh I bet wow you can see when a Twitter account's created all right let's go get my user ID this is too mad I guess it's too meta all right how do we get my user ID uh user by rest ID variables it's my Twitter ID well I have a long user ID probably because I'm not legacy uh oh our status code is not 200. status Our God status code 400 what's a 400. 400 status probably fail my own interview oh that's a bad request okay we messed the request up guys so have you messed up oh maybe that 22 is uh actually in the yeah yeah it's in the percent okay get rid of the 22 no 22. we don't have any 22s on that sorry good oh look at this here we go president of com AI oh URLs display URLs this is pretty nice should we write a custom Twitter client and replace the Twitter client with something that is fast and lickety-split we can now that we have secret access to the Twitter API right like there's an API and then there's an API right and that's like it's sad when these these apps do this right they have this whole developer API now maybe it is the same API and I'm just an idiot and search for graphql and just their search is bad um but all right let's see how can I make a Twitter API getting started access the Twitter API okay okay okay okay okay okay seems very complicated I tried to click sign up and they asked me if I was gonna like sell my data to governments or something and then I got bored so I stopped doing that like I'll show you what happens if you click sign up so these these endpoints don't look anything like these endpoints um which use this graphql thing now what's interesting about the graphql is I'm not really sure what this is so when I look at it it looks like base64 encoding but I tried decoding the base64 now we want to decode like this website probably get so many good leaks and I just get this which is kind of useless um so that might just not have a usable thing but we have user by rest ID with safety mode user Fields with super but all right so you basically make this request actually do we need features what if I just send variables does that work status code 200 no we definitely need features do we need any of these features to disable that feature 400. oh I think if we still get a 400 we can maybe get some text yeah yeah okay the following features cannot be no oh great um foreign one of my most accessible streams yet because I'm not like doing something that's like crazy hard I'm doing like hack this box level stuff you know all right good you're getting 971 viewers um the following features cannot be not we absolutely need a responsive webql graph timeline what about verified phone labeling can I disable that one no no that one can't be no liner so can I disable these that's a variable that's not a feature query violation variable has an invalid okay there's probably going to be some way there's some way in graphql to request like I don't know if variables and features are like a normal part of graphql I'm honestly not that familiar with it there's probably some way to request docs man whenever you can get things to document themselves that's the best all right let's just search for graphql Twitter API maybe someone's just documented and Twitter is public API [Music] did I caught my graphql in here oh okay maybe this is just then the Twitter API notice how this looks different though these are Twitter I that's 1.1 right this is Twitter API 2. so maybe this is just Twitter API one well actually you know what a better thing to search for would probably be user by rest ID oh here we go okay so I know this is not the real API if the first hit is uh is the twint project an advanced Twitter scraping and open source intelligent tools while evading most API limitations oh then they throw it in some elasticsearch no authentication no API no limits um wow wait this seems pretty cool why are they archival yeah so let's get some more graph let's get some more high quality uh graph URLs oh tweet detail here's another one all right so I'll notice that this has a completely different whatever this is it's completely different um these are my rest ID tweet detail really good list of these somewhere let's at least uh let's at least clone this and take a look at the Clone did the lighting get improved so maybe they have a list of us and maybe like if so someone's already done some work reverse engineering this so no it seems like it's interesting that this is different and they're still using okay there's two of them authorization and yes token so that seems to be important I wonder if that's an older version I wonder if we can change that and still works query violation variable with highlighted label I won't maybe we need that one back interesting okay so this is a different version of the API so that's like encoding the version probably just some hash thing I don't know let's um we should try a little harder at that base640 code I bet this is some normal graphql thing can does anyone know graphql and could just link me something to it uh hex down a64 code let's look at the bites the bites Never Let You Down uh what is it oh it's like b64 decode incorrect padding okay so it's not even valid base64 I don't know I'll throw some equals there we go I don't know that could be like AES or something right like whenever I see whenever I see 10 random looking bytes there are those bites random and they look pretty random regardless it doesn't matter uh we can just get it from Twitter so what else might we want to do we can get details about a tweet uh we can get user by rest ID user by screen name how do we get user by screen name that's interesting idea viewer here's another one these are secret urls you know I really don't like when they have a completely different API like this this should be refactor when they have a completely a good and this is something we do at comma we use for the most part the same API internally as our apps use right and if you're not using the same API the problem is it's just stupid to maintain these two things that are basically the same right and you're like well okay but the internal one needs more access well first off this one is an internal but you should always have your apps accessing through your apis um one of the things that I'd really like to see with Twitter is a better API and to again allow alternative front-ends right if Twitter moves to a more to a Less ad base than a more monetization based thing you should be able to plug the data into anything what you're paying for is access to the back end look I'm happy to pay like I think a lot of people are um you and your wife are so cute well thank you uh would pay 99 cents a month to watch these as long as they're in 1080p um you hear about your eight dollars uh uh might want to censor the authorization header I'm telling you it's it's junk I just this is literally a virgin Chrome browser um convert curl code to request code like magic oh interesting I'm just reading all the the comments the dev platform is under utilized by content creators yeah because it's confusing um we posted YouTube I don't know here here's another one Twitter ql user tweets and replies how come they're different for all of those all right so maybe we should have a goal W you're selling the voice in my head when I'm scripting yeah all right I'm supposed to talk right I'm a streamer is this recorder streamed on YouTube and then some cryptos cameras are great um wow a lot of trash would you are these real trending topics could we get real trending topics back you know I was the top trending topic in 2010 Black Rain was the top trending topic I'm hoping like the old days of the internet are kind of gonna come back when it's less about you really do have to wonder what all the people at these companies do right you just you just have to wonder like I look at all this stuff and I'm like yo like I don't know maybe it's arrogance but I think Twitter could be built and maintained by 20 good Engineers right um maybe there's something I don't see you know what I think we've we dove into the front end and off uh we kind of get how it works um we can go and a bunch of people have already done it like but I think I get what it is so it's this graphql stuff um maybe we can make some guesses that then it responds with some JavaScript is there some way is there some way to query like graphql like to dump the type system of graphql I'd really appreciate that um graphql verbose okay so what we get interesting things when I like let's take out user ID query violation right like so this leaks some information um yo you do those like webpacking.krs right I think actually what if I what if I comment off two of these does it complain about them both or just one no it just complains about one first one code let's look up tracing Trace ID um I'd like to intercept all right learn uh all right so quickly our guys reply do you want to see me go into backing stuff or do you all want to see me try to learn scholar exploring graphql I feel like I've done these before and there's like pretty cool ways so you guys I'll check the Twitter in like five minutes we'll see what the replies are you want to see me do scholar tutorials or you want to see me uh go into the back end and try to make some like numbers about okay how big does this thing actually have to be I read some things where they had like they were talking about like a Hadoop cluster with 10 000 nodes I mean it just seems wild to me it seems wildly over engineered for and again maybe I'm talking out my ass but like there's not that much there's a few things that make Twitter simpler than a whole bunch of of the other services right um so first like if if you're dealing with something like YouTube um YouTube the storage requirements on YouTube are insane right the the uploaded data being uploaded to YouTube is is is is petabytes right we're going to do this a little at least um so let's talk about the back end a little bit uh so let's say that there's well let's just look it up um number of tweets per day there's over 500 million tweets sent per day okay so we have 500 million tweets uh let's say with metadata it's 1kb per tweet all right so this is 500 gigabytes per day this is not a lot of data right compare that to something like YouTube um now to be fair probably one tenth of the tweets have have uh have either video or images embedded into them um those can pretty much go straight to your CDN you can have like a separate system for this right so let's let's look like I'll show you guys when you have an image posted on Twitter like um you know this truck here uh let's grab that okay so it's on pbs.twimage um this looks to be where all the images actually are images uh JavaScript fonts so that's ABS maybe we can figure out what these things stand for and then video is hosted somewhere else um can I do something like format equals PNG does that actually give me a PNG let's take a look uh what is this yeah content type image PNG cool can we get a BMP let's see if we can get a BMP oh let's put that in quotes you know like lead hackers probably have like better ways to do this than like holding down the back arrow but did we get a BMP nope we got zero bytes bmps are not supported 404 no BMP for you can we get a chip let's see if we can get a chip oh we got a Jeff but it was actually a PNG okay so the gifs are alive guys just like okay the gifs are alive um let's see if we can do small Mall what if we smell small wrong oh can't be found okay so they have some image ingestion server that's taking these things in storing them seems great resizing them a whole bunch you know uh that's good so that's what you're doing for images okay now 70 million 500 million tweets okay I mean that's not nothing okay that's actually more than I first thought [Music] um here's what they talk about their their their back end so like I said if you want to understand the back end of a company uh the best thing to do is to read the engineering blog uh you'll at least get some some basic ideas uh so these graphql I mean I just want to like get an idea and some of this uh you know it's hard to tell from the outside but we can try to figure it out um like so we're making this graphql request right and then somewhere in the back there's a database right and there's a question of how this request actually gets routed to the database um like I said I'm not logged in here also I mean maybe here let's try a search and let's see let's search for uh Twitter all right and then what requests actually happened user by rest ID you know we know all about user by rest ID what does the search actually happen search is an interesting one right the search is probably handled by a completely different system um than this I mean so so far what what requests have we found we found tweet detail and user by rest ID so those are probably pretty direct access to some kind of database um now from this blog post they said they used Hadoop and Manhattan Manhattan is a custom internally developed distributed key Value Store um I mean I'm sticking with what I said about the 20 engineers um I mean okay there's not there's not maybe 40 right maybe it takes a bit more because you do have the thing that makes Services hard is when you think about how um horizontally scalable they are right the more horizontally scalable of services the uh the better it is right think about something like WhatsApp right WhatsApp is almost perfectly horizontally scalable right if it's just if it's just point-to-point messaging right because I can I can Shard all my servers and then like okay it's got to find The Shard when it delivers the message but that's really no big deal right whereas Twitter needs to do something like and this is kind of a difficult request you can think about how it might make it is when it makes your feed right making your feed is a big is a big question and this is a distinct thing from search right so yeah like like that there's some there's some difficulty there um they've all switched also to these algorithmic feeds instead of just straight up linear feeds um I know there's a lot of controversy around that I generally think algorithmic feeds are better um it'd be nice to know what the uh the algorithm is foreign let's talk about their event processing let's look into what their event processing stuff is this is also recent um these things do change but not that quickly okay we generate approximately we approximate 400 billion events every day so that's pretty far off from the 500 million tweets again so they're saying 400 billion events okay you're also going to have to deal with lights and stuff right um so you can think about how that might be represented in something like a table uh there's various event sources we consume data from and they're produced in different platforms and storage systems such as Hadoop vertica okay I've never even heard of this one [Music] ah [Music] vertica analytics platform what does it do analytic database management software company that's too many words man column oriented okay I've heard something about this so his date is by columns rather than Rose okay well they use it um Manhattan distributed databases their custom thing so Hadoop means a lot of things too Hadoop is a is a like really large um thing but I I assume when they say that they have storage in Hadoop they mean hdfs all right hdfs is hadoops uh storage system I remember also reading that they were storing things in in Thrift um I haven't seen Thrift since I worked at Facebook uh but Thrift is um Thrift is uh similar to uh it's kind of like protobuf but I think it also includes uh like a IPC system like uh maybe even via RPC it's some some some it might even have an RPC built on top of it right you can think of like a uh there's like look like zmq if you want to like understand zmq's docs are pretty good to understand any IPC um okay Kafka this is another it's Apache Kafka open source event streaming platform this gets into this gets into fancy stuff um at comma like so we we wrote Our Own um we wrote Our Own uh distributed file system because they were all just so complex and yeah I mean when you do deal with these very complex things you end up needing like multiple Engineers just to maintain them right to maintain this large cluster and deal with you know who knows how it might go down I'm surprised that things are as robust as they are but uh so this is what we use internally to store our petabytes of uh video data okay the old architecture is shown below new architecture on Kafka and dataflow yeah they said they were talking a lot about how they moved to Google Cloud oh yo guys I got a vape by the way vaping is my new hobby um okay foreign events okay so what does this say about how they're doing their events so I like got distracted this was like confusing new architecture real-time pipelines on Kafka and dataflow um patch compute Clause latency what's heron Heron Therapeutics all right let's see let's read the uh comments if you are wondering about those IDs they're just string constants probably for versioning wait oh that's a really good idea wait why did I not just look in the in the in the yeah I can just look in the uh we just look at the scripts um that's smart that's not internal you're not internal to Twitter doing that are you uh so where are the script shows did ABS twitter.com responsive client web Legacy um mean JS yeah that's a good idea so we'll just search in here for user I rest ID uh oh is this a pretty fly here is there not so what this does can I pretty fly this how did you get yours to be pretty um uh online JavaScript beautifier paste in that huge beautify code of course it's written in JavaScript take forever um oh we have all the feature switches there too that's cool that's really slow create a new file called main.js um I can do this in here kind of not yo for Ross man he's everywhere uh okay we're gonna plug in pretty fine vs code pretty fi Js uh how to beautify a JavaScript file on Save so true beautify oh I need this you pretty five Json now I want JavaScript formatter prettier code formatting using prettier that sounds good a lot of people use it command shift p okay command shift p pretty [Music] prettier create configuration file no command shift p select the tasks you want to want which is command command shift p pretty far let's create configuration file no code Snippets for JavaScript no prettier formatter prettier code format I I don't understand using command palette a format document oh God I can't read format document there are multiple formatters for the JavaScript one of them should be configured all right it's doing something I think there's a bar there moving did this pretty find my JavaScript I don't know what editor you're using um live streaming myself coding had to suck for the viewer but if the encrypted responses and protobufs well what's this elephant bird is Twitter's open source library of lzo thrift and or protocol buffer related Hadoop input formats okay well good um scaleless sound Scala sounds more fun and hardcore we'll do a little scalato oh the prettify oh press the button oh yo let's go what I was wasting so much time let's let's go let's go in here okay press this button oh that was so boss bro I'ma like that tweet yo Yara slav coming through coming through with the coming through with the legit yo this shows you I don't know anything about the front end right all right I'll use her by rest ID oh look this is so nice Bros who wants to send user by screen name let's go let's go Elite Twitter hackers boys we're Leaf Twitter hackers um user by screen name now let's do that now it's gonna bitch and maybe like yo but you you can't use that because you're not sending the right thing so then we're gonna send the right thing okay variable screen name has an invalid value all right well let's use screen name screen name is realtor shots is that going to be me oh look at that wait wait wait wait wait wait wait wait now does this API give us Elite super user access where we can see band people Twitter suspends accounts that violate the Twitter rules I'm sorry real Donald Trump you still can't be accessed even through the API yo yo this is that though hey look oh we got all this stuff oh cool yeah oh look at this this is this is the elite list let's just copy and paste this list this list looks so useful moderate tweet can we moderate other people can we edit tweets Bros did we just find the secret edit button okay um let's copy and paste this oh we want stuff to be ad free we might be able to do that okay so we found the secret Twitter API you know they talk about how this is a secret good open pilot well that's not real but the secret good Twitter API exists let's just call this api.js and paste that there okay cool um prettier was a scam all right what do we want to know can I like filter all just the ones that use the word operation name because that's all we kind of care about right there's probably like ways to do this if I actually do JavaScript let's just look through and see if there's anything cool sorry for edit oh that's cool check tweet for nudge Community by rest ID Community reorder rules quest to join Community oh this is the secret secret Twitter API guys secret Twitter API wait a second Bros should we downloaded three create tweet down vote I wonder how that works delete retweet delete scheduled tweet DM muted timeline edit draft tweet edit scheduled tweet it doesn't look like we can edit a normal tweet um we can check followers what else we got we can see noteworthy accounts you can pin replies okay that kind of makes sense we can enroll coupons we can remove a follower should we remove a follower what follower Would we not like a little safety mode settings that sounds good um trusty friends tweet edit history Twitter article delete all right you know what I just want operation names uh I I know wait what am I thinking you guys have I spent my life on the text editor for so long that that I forgot how to grab shame on me let's grab the rap yeah now we're talking put that into uh API on Dot JS yeah here we go oh look it look that's so much nicer you won't even throw that through sort let's throw that through sort guys I love pipes I love the command line all right ooh Bird Watch create appeal create notes um you can't edit tweets you can only edit schedule tweets and assume favorite means the same thing as like likes favorite tweet unmoderate tweet can I moderate tweets well I can moderate tweets all right well this is the this is the secret Twitter API so uh the secret good Twitter API is uh is here cool um ah let's go back to where we were uh thank you thank you by the way to those to the people who uh who sent me that uh I'm gonna be right back bro we got 1200 viewers all right so I think graphql let's just do a little more searching to graphql I don't think it specifies anything about a database so I don't think it actually tells us anything about the front end let's just look through the comments maybe we got something good all right we have 1300 viewers oh that's cool encrypted response it's not encrypted it just looks like nothing like it's just it's just a version this query ID thing doesn't seem like it's really anything maybe good um ooh graphql introspection oh graphql allows us to do this what by querying the schema field always available on the root type of a query oh how do I do that I don't know let's try it okay what if I go schema there instead 404 uh this would probably help if I like understood what graphql was graphql introspection how to should I search for four Noobs sometimes you add for noobs they help you yeah the graphql playground it's like a request no no I don't know if this works true I don't do anything foreign fetch existing types schema query type oh do I put schema here I don't know I'm just guessing I don't think any of this is right I don't know how graphql works oh what is this type name user is that the skin no I didn't do anything it's always just says that okay whatever you want to First commit is changing tab space for two from two to three yo if they use three spaces I'm out I'm out I can't work at a company that uses three spaces that's impossible um pronounced Jif uh imagine if George was down to be a technical co-founder why would I found a company with someone non-technical right uh does he have access to the code base no I don't have shit uh yet another Elon fan joined us uh I would like to see you get root and replace the blue check with an icon determined by an hourly Twitter pool should we get root um wait how do you live stream to Twitter you got to be late like me oh yeah I don't overreact Iran you know it's like it's bad thing there that's cool 400 bad requests if you ask me I I did kind of know that um okay what were we gonna do oh yeah we were just talking in general about the back end okay they have 400 billion events going through Apache Kafka Crossing billions of events in real time day versus we consume data from different storage systems I already read this Kafka topics event processor Cloud did they really just move everything to bigtable and data flow why did they do this but processing and relay event processing which converts Kafka topic events to Pub sub topics with at least one semantics I'm going to use these things first off they love Java a lot of this Apache stuff is like deep in the Java land we wrote Our Own pipeline at comma 2 because I was sick of the complexity of all this we have like a dead simple uh pipeline for processing all our and all stuff um and then oh we wrote this thing I wrote this thing the end of last year called mini Ray mini Ray is is like Ray but mini and it just uses our existing pipeline it's basically like you can run a chunk of python code across the whole comic compute cluster like nicely uh I was pretty happy with that I spent like a month writing garbage and then like I wrote that it was 100 lines and it was good um all right let's do a few scallop tutorials and then uh we'll uh we'll call tonight all right hacker rank Scala Scala basic view all skills Gonna Learn Scala guys basic fuels what okay I'm gonna create an account hang on I'm just gonna create a throw away all right I'm here to prepare for job interviews I'm a working professional uh I've been a professional developer for more than 10 years I have my interview in a week what about tomorrow okay guys oh okay okay print hello world I'll make this big no no we can't use closure guys we gotta use Scala Scala okay def F print Ln hello world submit code did I pass I think I passed okay next challenge let's go uh print hello world end times oh okay this will be the outline of your function body okay uh do I put this inside the object that it's solution that extends app uh def f is an INT okay maybe we should read some Scala code first Biz Buzz Scala all right all right that looks doable okay one until hundred map oh man okay wait okay guys I don't know if I'm allowed I'm cheating right now I'm cheating on my interview right now I'm using this okay we're going to try it out okay people are like George doesn't actually know Scala No One's Gonna respect me at Twitter all right so we need to do a for Loop so we're gonna do four I'm gonna get the tap there for I equals zero two n okay we gotta print many hello worlds right so we're gonna use hello world and forget how I print things again in Scala print Alan hello world all right did I write good scholar uh oh I did not write good scholar inspected output and what did my thing output not all all wait unlock this test case for five hakos he compiler message wrong answer oh is it because skull is one of those languages all right guys I didn't use any hacko it's great yeah guys okay give it a list repeat each element in the list and amount of times okay oh where's my where's my template wow we got viewers all right people are liking the skull like watching me not be good at school okay um this will be the outline of your function body oh okay deaf f uh okay four I oh no don't tell me don't tell me wait wait can we can we install Scala on my thing so I can like play around a little bit can I get like a scholar prompt Brew install Scout root install Scala jvm base everything gets built on the jvm man all right so how do I use Scala [Music] maybe we should do some Scala tutorials first is Twitter really written in scholar or am I being trolled scholar school started as a lecturist to prepare experienced Engineers to become productive scholar programmers oh look at this the first few weeks will cover basic syntax and Concepts bro I need something tonight oh good good good good good good all right great I'm glad that it depends on oh my God we're installing x11. uh uh okay it's Carla you can give the result of the expression and name hey we already did two hacker ranks okay so we're Elites hacker rank scallops um you can create functions with def well that would have been nice to know before I didn't fail at creating functions and create Anonymous functions that's cool you can partially apply a function with an underscore I see add two on all right cool you can carry the functions [Music] stand fill in the first parameter and partially apply the second but why can't I what if I just don't do that what if I just set it to that it's the carrying I know I don't need that extra thing there partially apply the second what Curry dad wait what it's installing python 311 is that the new python where's python 312 I'll probably twelfth now oh my God guys we're installing the whole world here guys all right 2 000 viewers let's go um yeah Curry functions uh effective scholar oh well okay there's no such thing as effective skull that's good scholar line documentation or in Scala install Scala on your computer the SBT bill is this done yet no it's not done we have to pour the add weight out icon theme okay let's come back to here now that we now that we've learned Scala let's learn let's apply what we've learned uh okay how do we get the length of an array Basics continued lists Maps okay numbers array let's preserve order are these lists yes the lists okay good um Scala probably wants us to use a map here functional combinators zip partition I'll flatten okay okay I think we got this all right a flat map sick bro that's exactly what we want okay I'm gonna write this as a Scala one-liner because we're neat programmers ready oh if this works if this works okay ARR dot flat map okay let's figure out what we have to do here the outline your function body fill in the blank part what's numb s is the number of times you need to repeat the elements okay so we're going to do arr.flat map um lists okay so it's being called with this someone just do nothing so we need to like multiply I really know how to do that well okay hang on hang on hang on what if I say X is not a list but I can create a list can I not um man if I just wrote something good it would have been mad skull skills but you know I don't have those guys goes okay flatten we we do a flat map but we want like a like in Python it would just be this but it's not python Scala list Scala uh insert multiple nope okay maybe we need a loop it's a for Loop something that we can like there was some of those in those in those examples I read what are the examples um Scala Loop Scala for Loop okay uh there's other ways to do this though Scala range array is what I really want on your creator array with range grade um oh there's something called range I mean okay this is a bit of a hack but I think we can do range one dot and num map X returns num okay let's see run code compilation error error not found value range oh we need to import array underscore wrong answer but it's almost the right answer we just outputted oh no no we don't want num there we want y implies X so let's go let's go is this some scholarship all right come on come on okay wrong answer your output um expected output okay that's the input your app one one two two um maybe we have to put zero there I don't know if that's inclusive with num or not no it's not inclusive okay oh yeah yeah buddy let's go yo if that works yo who's the king of Scala this guy this is right that's right King of Scotland man I'm ready to go beer I'm kidding you know don't worry I'm not really that arrogant guys I'm not okay um for hackers using Scala all right well we're a hacker using skull this will be the outline of your function body okay your only implementation of a filter function we recommend not using the inbuilt library function well let's see what scholar School tells me filter oh this looks good let's use filter okay uh we want to filter things that are less than x so let's just use I don't care what you recommend we're just going to say ARR dot filter and then you filter X uh uh filter it if let's try to do this dim filter we want to return true so uh why one two and zero are the list elements oh they're less than three they're displaying the order they were in the original list okay um we want to know if x is less than to win does that work yeah let's go let's go all right these are getting too easy guys we need to we need we need some more Scala we need some more complicated skull let's go am I going am I ready to pass a Twitter interview can I write Fizz Buzz man I'm just gonna be the one of his lists of pairs of functions all right uh we we oh I think we're done there's not even another one guys we finished all okay great we've written everything we need um oh these kind of look hard oh we have skull in it was it SBF SBT close SBT not found how come I don't have SBT one plus one yo that took a long time guys see how long that one plus one talk oh Varney well Steve Res three cool love Scala all right good I think I think we're good on Scala that's probably all the scholar we need to know um all right yo we have so many people here I don't know if we should uh we should leave the live stream but it's it's uh it's getting close to my bedtime let's let's read through the timeline and uh see what we got let me drag this over here this reminds me when I was trying to reverse graph ql code with um he nice tea dunk using common list amazing things are happening but the media will be silent bro who's the media what when when you want to know about the FDX store you can go to the New York Times or you could go to autism Capital who tells you everything about it um can you access the endpoint blah blah blah blah graphical introspection here we try that uh full thread on learning Scala oh well Scala school is the top one okay cool um you should make a timeline way back machine on orbit and then yeah yeah uh relevant context best streams on the internet thank you thank you thank you um Brew install [Laughter] uh I get it I get the joke uh um reload vs code oh I see format document yeah sorry I'm an idiot um resolution approved that I'm glad about graphql client oh that's an interesting idea uh finally someone to fix the lights thank you um Doge uh thank you for that guy prettier uh we looked at that admiring on us work some nfts those are for persisted queries oh I say how do you go live on Twitter you got to be late uh uh thank you I'm glad you're entertained yeah I'm just reading Twitter you guys are probably reading Twitter too Elon Musk come watch Miss while he's watching uh uh now you don't got better things to do than watch me stream I hope what's the code editor I'm using oh great okay okay nothing too great I am interested in the quote tweet still oh boy I will miss this place Twitter stream on Twitter exposing Twitter code guys I don't have any special s all right we learned Scala that was pretty good let's think a little bit more about the back end I know we'll wrap this up by 11. I'm sure you guys all have jobs which you drive to in person uh tomorrow all right so the Scala thing was a was a diversion but you know I did get a lot of views and that's what matters yeah you gotta do it for the people oh okay so Hadoop and our current distribution is Hardware shown below I like to the new approach okay it seems like they moved everything it seems like they moved everything to Google Cloud um which is interesting I mean yo Elon here is where the real money is going to come right see I'm going to work for 12 weeks and then I'm gonna be like well I can lower the Google Cloud bill by 25 million dollars right um nah look honestly and I'll say this um I'm gonna say the same thing to uh uh at my interview Elon lost how many billions on this it's not what it's about right um I'm here to help the cause if I can you know I'm not here to help the cause forever but I think it's kind of like you know some people go out and like canvas for a political party uh that's kind of how I see this I kind of see this as me uh you know look if I can be useful if I can't be useful I don't know you know maybe maybe uh you know I get the Twitter and it turns out that it's all just uh you know the people really know what they're doing and I'm honestly not that useful I don't know Scala uh um you know I don't know Scala I'm not that familiar with all of these really big distributed back-end systems um I don't know react uh but you know I can do what I can I figure things out pretty fast uh and you know again it's really it's about the cause um it's about the cause for a place for free speech exists on the internet and uh you know what's something that's really important to me there's not much in the world that's really very important to me you know a lot of things it's a joke but I mean this kind of isn't right I look and then it's one of many things it's always going to be uh like an ebb and flow right it's always a cat and mouse game there's always the people who want to make you know want to restrict speech and who want to make speech free and the internet kind of goes back and forth but I felt that the internet's really kind of sucked for the last 10 years um or at least it's gone downhill now maybe not in the last 10 years maybe it's really been like the last five and I think this is something that could potentially really reverse that tide fucking health I'm down to uh if I'm useless then I'm useless you know but uh that's that's yeah data transfer in Manhattan using Rock Steady I think roxyb is something kind of like level baby persistent key Value Store okay it's okay Manhattan is Twitter's internally developed key Value Store it is the default storage solution at Twitter for persisting real-time Servicing it's used to serve all or nouns at it's like it's interesting right there like like look just you can tell a lot by like their terminology right like they have nouns that's actually a really cool way to think about it right like nouns and verbs um data transfer between storage machines is a typical process when we started from our older okay yeah they talk about they had these custom uh storage engines but I don't think rocksdb is multi computer I think it's just one computer so it's interesting it almost seems like Manhattan is backed by rocksdb now um which is cool yeah they had they had these custom storage engines so I would imagine that Manhattan now has a roxdb storage engine I remember reading about these in one of their engineering blog posts yeah Manhattan had two read write storage engines this one a read one and a right one most mhb eks implemented as a my sequel plugin a plugable storage engine yeah okay great um SS table storage engine tunable amplification cool uh right amplification is also done it's also done by uh by Flash you know if you're a flash if your flash is like really full you have a lot of right amplification with it moving things around you'll burn it flashes fast um our our head of infrastructure.com is always talking about he loves talking about like the lifetime of these ssds and stuff um we only ever had one group I can get into that but um we Kickstart evaluation of roxdb as a new storage engine this is also what LSM based oh log traction merge string okay I kind of know this um with confidence that it could match box TV storage engine great Manhattan is a multi-tenant system that is a single cluster shared by multiplication and data sets ah Okay cool so it seems like it's this big distributed Twitter Manhattan's not open source right I wonder why not [Music] um moving from a monolith to microservices migrating to multiple data centers using Cassandra this is another one of those Apache things [Music] um okay but it seems like most things so I bet you most of these these queries that we're doing are just hitting so there's this thing Manhattan which sits in front and then in the back end there's a roxyb okay and roxyb is a lot like level DB if I recall correctly I used it once um so this is like multi-machine and this is single machine again a lot of these uh things can be sharded so it's used to serve okay so all four nouns so when we're getting nouns like when I run that out when I run my little script here uh which gets the noun of me right here that gets the the George hotz noun I would assume that it's hitting Manhattan and then rocks DB right so there's probably some cash in front of it and maybe we can try to get some insight into what that actually is uh let's throw some V's oh this is the big curl again I don't want the big curl let's do the little girl little girl I'm throwing a demo okay um so you're gonna figure out what server this is uh this is the response from the server server tsaa okay so tsaa uh so like this is web front end Manhattan multi-machine then this is determining which machine that the George Hots things actually located on and then it's sending that um there might be some caching layer in here all right so usually usually you throw a caching layer after your web front end so you don't have to hit the database if I make the same request over and over again oh it might even tell you something Twitter response tags bouncer compliant gsaa how do I disable HTTP 2 maybe I can uh how do I ask for oh tell Carl to accept okay there we go at least it's 1.1 now um wow I don't know there was http 3. cash that's some perf header uh it's a Json wow that's crazy that sends that [Music] um okay and then like let's go back here like is it all that stuff when I like load this what's actually loading the timeline and then is it actually requesting for each tweet I feel like not all the stuff showing up here right because like where's my where's my JavaScript I don't understand clearly not everything is here oh is it does this show up as part of tweet detail it might authorization denied by Access Control oh I can click there and format oh that's nice yo by the way whoever showed me that thank you I love that um I don't understand okay why is something's filtering here because it says 26 out of uh what's that this hxhr why is it not showing all the requests this is this is where I like having the live uh Chrome Network isn't showing all requests because then everyone in the logs is just streaming at me filter is red which means there's something in the filter search box that doesn't match any of the results you just can't see the search box oh I have the word API in here oh that's dumb right there we go and here's okay here's the JavaScript being loaded at the beginning oh that's great can't believe I just put the word API in there okay so here's all that JavaScript being loaded um whatever these client events are I think they're being blocked by my ad block uh we can turn it off but who cares and it looked like the other stuff guide dot Json well that's the trending topics sorry if I have to get close to the screen you guys definitely can't see these are JavaScript fetch xhr to buy rest ID guide.json search for this switch in here oh that's not right um sync message like four what's this what is that two okay this guide that Json seems like a decent part of the API let's see if we can find any documentation on it um all right well ah yes all right so it seems like this authorization guest token thing has been around for a long time um I wonder what guide is I don't tell you even if I could speak Japanese I don't think it goes into it um all right guys it's 11 o'clock it's my bedtime uh if you don't watch I stream on Twitch frequently but you know I figured it was uh it was more appropriate to uh to do this stream on Twitter let's just make me big uh my name is George Hots uh uh I stream I tweet nah man I don't know look when you know we had no idea what we were doing but an artist's first work can be his greatest uh Ode to the night rats Club I think I should go listen to it um yeah thank you for watching uh hopefully it'll be entered Twitter and I'll see what I can do to uh help the cause uh you know I'm a old school internet guy I want the internet to be as uh free for uh you know my children as it was for me right so anything we can do to help that cause that's what we're here for all right thank you all for watching have a good night bye gotta make eye contact