Sign Language Recognition - Live Coding & Data Science

Key Takeaways

hello hello hello hello it is February 23rd 2023 and we're here tonight to do some live coding to hang out to get to know each other and um generally just to enjoy what's the fact that we're here on on this Earth so let's go ahead and do that hope you're doing well tonight let me get all my chats and stuff up and let me know if you're in the chat if you can hear me okay hey yo big Samosa welcome to the chat you're the first one welcome um we are going to take a look at some coding caggling stuff what do you guys think about that Naomi Haiti Gabriel everyone's in chat hanging out we got three people on Twitch by the way if you're not already watching on Twitch please make sure you do that also I will put the link in chat that's where we hang out although YouTube's been showing up a lot more recently what am I gonna do what am I gonna do nothing I can do about it I can't control you guys I can just keep on doing what I do all right let me make sure I have my windows up and not listening to myself on repeat and um yeah this is this gonna ask for my authenticator app token I reset my cookies on my computer and now I gotta re-log into everything authenticator app this is what I call two-factor Authentication let's put that in remember this computer for 30 days that sounds about right um Creator dashboard that sounds about right and let me show you guys what I'm looking at make sure I got this um the right size there we go the infinite Loop there you guys are that's our twitch and let me make sure I have YouTube up YouTube look someone's Live on YouTube that's us moderation activity is Now hidden I don't know what that means all right let's go ahead and get to it tonight um here's the deal on kaggle.com there are competitive data science competitions that's a little redundant to say competitive competitions but let's go ahead and look at all of them there were some new ones released that I have I honestly have not looked at I've not looked at any of these but I want to take a poll and see what which ones you all might be interested in taking a look at and then we could see what what they're all about what do you guys think let me make sure this is set up correctly yes hello from Donna Texas Donna Texas where is Donna Texas Donna Texas oh right down there at the very bottom of Texas that's where you're viewing from Welcome to the chat thanks for hanging out all the way down here in the location of Donna Texas let's go ahead and um let's not ruin everything I have my my second screen is not here so I'm just complaining today it's a lot of complaining for me I'm gonna stop I swear um what we are going to do now is um is there any spatial analysis I don't think so so here's the new ones that that just launched if you want to join the the uh poll I guess we'll call it join us over on Twitch and I'm going to go ahead and start this twitch poll so let's um let's get this over here I'm going to go to this one I'm going to start a poll this is called pulling back the curtain clip that no manage a pole new Pole which kaggle competition should we should we we look at okay which one should we look at um sign language maybe you should actually look at these a little bit before we make the decision because I'm making you all decide so sign language um goal of this competition is to classify isolated American Sign Language science this seems really cool you'll create a tensorflow lite model trained unlabeled wait we will create a tensorflow lite model trade on label Landmark data extracting media pipe holistic solution thank you uh your work May improve the ability of Pop sign star to help relatives of deaf children learn basic signs and communicate better with loved ones I mean why what's better than this seems like a great idea hey j-rod thank you for subscribing by the way uh spin the wheel for you spin the wheel for j-rod thanks so much for subbing What's It Gonna land on oh no type pizza oh so close to doing the push-ups I really don't want to do the push-ups uh so let's let's put this here let's type Pizza 10 times can you hear it see it one two three four five six seven eight nine ten there we go that's for you all right so sign language is the first one we're gonna look at or the first option that you can vote on um amp Parkinson's disease progression prediction use protein and peptide the goal of this competition is to predict MDS updr scores which measure progressions of in patients with Parkinson's disease the movement disorder Society sponsored revision of the United that's a long acronym is a comprehensive assessment of both motor and non-motor symptoms associated with Parkinson's so let's see what the data looks like for this one data is only 59 megabytes how is it so small that's what she said um who is that did someone just subscribe again I can't figure out all my windows yes thank you for subscribing S1 lvrp um so this one is uh we're gonna spin the wheel for you man I'm just all over the place can't get in a Groove um what's it going to land on no yes take Pizza 10 more times I swear it's not ringed rigged one two three four five six seven eight nine ten more for you thanks for subscribing um so the next one is this ones Parkinson's prediction I'm surprised that this Parkinson's prediction one is so small 59 megabytes and it's only CSV pi and so uh so the goal of this competition is to predict the course of Parkinson's disease using protein abundance data the complete set of proteins involved in PD remains an open research question and any proteins that have predicted value are likely worth investigating further foreign the core of this data set consists of protein abundance values derived from Mass Spec ometry readings of a cerebral fluid samples gathered from several hundred patients each patient contributes several samples over the course of multiple years wow this is a cool one too um interesting so this is just tabular data I think so we're gonna have to learn about this if we if we choose that one so that's number two um hello everyone in chat by the way silver hi Rob gonna seize yo living that data science life on a Thursday night Herbie Hoover knows what's up uh do you think that pullers will take over pandas probably not completely or if it will it'll be like a slow burn but there are a lot of competitors to pandas it's not just polars out there okay so another competition that's out right now and this could be an interesting one although it's kind of every year um is the NCAA March Madness one so um March Madness doesn't start till next month um but they're I think just launching it and it I haven't looked at this yet is this evaluation is where's a metric oh they changed the methodology is change from prior editions of this competition submissions are now valued on the prior score between the predicted probabilities and the actual game outcomes Breyers score by the way people um I know they spoke about this a lot in the book super forecasters which I really liked if you need a book that you're interested in that you think would be fun to pass the time reading check out the book outliers or no super forecasters but the Briar score was used a lot in that there are these professionals forecasters who would use it um it's equivalent to mean squared error in this context so they were using log loss for about 10 years or maybe longer and now they've switched to the Briar store so that would be number two number three and then I think there's another one that launched that we could look at that is um all competitions we're going to skip over the playground series one and look at stable diffusion so this is like a competition to do the opposite of stable diffusion how is the how are these so small too size is 3.24 megabytes so we're gonna have to dig into this why why the data set is small for this too let's let's understand and agree those terms maybe they just provide a few images as examples so I guess you can create unlimited data by creating your own stable diffusion predictions and then you have to feed it back in I know that um I saw someone post on Twitter about this about how they uh basically use stable diffusion to predict to create an image then they use the image to create the prompt with their model and they just went back and forth back and forth until it reached a steady state with some starting things uh so they give it a prompt and they see how yeah how generic the I guess I don't really understand how it worked okay so that's the the third one so let's start this poll which kaggle competition should we look at probably bad English this is the poll in on Twitch if you are on Twitch right now you can vote on this if you're not on Twitch you should join us here's the link what do people want to look at March Madness is in first place we're going to learn about the Briar score if we check that out sign language is now tied for first place look at this this is so fun and in the meantime do you guys want to talk about anything else is there anything else going on in your lives I hope polars replaces pandas seems more ergonomic to me because says Adrian Adrian cool yeah people do like polars well stable diffusion is now coming out late to the party stable diffusion stable diffusion is really coming out there and the sign language competition too look at this there's such an even diverse excitement for all the different kaggle competitions out there look I can't guarantee we're gonna do well in these competitions but I can guarantee that we're going to look at the data and we're going to try to figure something out life advice for freshman software engineering major well I didn't study software engineering in school so I can't relate to you specifically I went back to school afterwards to study data science but it sounds like you're passionate about it and if you're not pick a major that you actually are passionate about that's a mistake I made my freshman year is I switched into something that I wasn't as passionate about because I thought it was like a better career choice and then I ended up coming back to computer science which is what I transferred out of my freshman year so where you vote going to where's my vote going to fiegel what are you asking this vote is on what we're gonna do right now what we're going to look at so if it ended right now stable diffusion would would win pandas will never die says Cho core latte yeah I mean it's like it's like a staple of data science it's never gonna completely go away all right if we get a tie here then I'm just gonna make the the Judgment call on which one we'll do but but you can try to sway me in the chat by saying something MMD is in the chat MMD always going to bed whenever I'm streaming always telling me he or she is about to go to bed never ends up doing it what time is it in your time zone 4 a.m 3 A.M um is the music too loud let's bring this down a little bit pandas will be incorporated for sure Incorporated in what sense like bought by a corporation I'm a guy okay MMD MMD is um always about to go to bed it's just what you do how do I so we see the time remaining because of this purple bar this was a five minute timer but sign language competition has come out they're strong I think the sign language competition not to get ahead of myself I think the sign language recognition competition just launched today launched 10 hours ago hot off the presses there's no way anyone knows anything more than me about it except for the people who launched it how we doing here whoa sign language is is taking a late jump ahead by three by three ahead of the stable diffusion it looks like we are going to look at some sign language stuff tonight the only downside of this the only downside people is that it just launched oh and then it jumped up to 10 at the end someone didn't want to be part of the losing team so they switched their vote all right we're gonna look at the sign language competition the only bad part about it just launching is there's no they're probably not a lot of notebooks out there that I can steal from so we're gonna actually have to do some stuff ourselves and figure out while you're waiting could you please check if this correct ml process I'm learning ML and ask a question SEC overflow um Yara I normally I don't do this but I'll upload this oh I need to sign in because that whole thing uh this is not a quick this is not this is like you're asking a lot of stuff here you're reading data train test split I think you just need to read up on this it looks like you're you're doing all the right stuff it looks all right but I'm not going to spend the whole stream looking at it um sign language competition okay let's go here sign language competition here's the data set data set description deaf children are often born to hearing parents who do not know sign language your challenge is to in this competition is to help identify signs made in processed videos which will support the development of mobile apps to help teach patients sign language so they can communicate with deaf children this competition requires submissions to be made in the form of tensorflow flow light models Google's coming in here on kaggle and making sure that we only use tensorflow models no can could you just take a tensorflow lite model and wrap you make it a wrapper around some pie torch I wonder if that's what people are going to do you are welcome to train your model using the framework of your choice as long as you convert the model checkpoint in the TF light format prior to submission oh so that's basically it all the models are are a bunch of uh floating Point values in and basically model weights in an architecture that is defined by the model either in pi torch or tensorflow or some other thing so what they're saying is you can train it on anything but you basically have to convert it into tensorflow Lite format in order to com submit to this competition I'm guessing because tensorflow Lite then could be used in a mobile app because it's a light version please see the evaluation page for details everyone's using parquet these days we got parquet files that's the landmark data landmarks were extracted from raw videos with the media pipe holistic model not all of the frames necessarily had visible hands or hands that could be detected by the model Landmark data should not be used to identify or re-identify an individual Landmark data is not intended to enable any form of identity recognition or store any unique biometric identification okay I wasn't thinking about doing that but now I'm kind of thinking about doing it because he told me to told me not to but I'm not going to don't worry I obey the rules uh do you have need to be good at math to do this testing probably a little good at math thank you just a question do you do Eda on full data or training data only or it doesn't matter you should if you're creating your own data set you shouldn't touch the test data at all you should have that just off to the side so you shouldn't explore that data I guess it's kind of hard to be objectively stratifying that out or whatever the best is if your test data is something in the future you can actually validate on okay so a frame is the frame number of the raw video row ID is unique identifier for the row type is the type of landmark one of phase left hand pose and right hand so they're using media pipe I think what they're doing is basically taking media pipe and giving you media pipes outputs run on a video so that you don't have to run that yourself which makes sense that's similar to a little competition I'm helping right now run right now for for kaggle in the NFL we gave like the helmet boxes already from the model output so if you don't want to have to create them yourself you can just create some sort of model that sits on top of that so look media pipe can already identify in most cases all these different landmarks of a hand so when detecting sign language I don't know sign language I don't mean to yeah I I should learn some sign language but uh it can detect your hand and then if they've already run that in all these videos they can grade it based on that um and you you don't have to run the processing for it I don't know if that's exactly what they're doing what am I doing here uh but it looks like they've done one for face left hand pose and right hand now it's a little strange that they have all these when I would think it's only going to be hands that we care about but maybe there's another reason maybe the face landmarks is the reason why should not be used to identify an individual because you could identify a person the hand landmark in Mark index Landmark index number details the hand Mark location can be found here okay so then they actually linked look the same site that we jumped to these are all the landmarks in the hand that there see this is a hand that they're using um that they're going to provide us XYZ is the normalized spatial coordinates of these Landmark these only columns that will be provided to your submitted model for inference oh okay so we don't get any video let's accept this this is cool this is cool so it's all parquet CSV and Json we don't see any video I don't think got it okay this is a really cool competition I think face is used in some signs oh it is like yeah yeah like you're doing it close to your face that makes sense okay so let's finish through this um they're all in parquet files which is awesome train CSV the path to the landmark file the participant ID unique identifier for the data contributor sequence ID is the unique identifier for the landmark sequence and sine is a label for the landmark sequence we're going to have to explore this in order to understand exactly what's going on but we're trying to figure out sign language from XYZ points of hands like this or in some cases faces so we're going to be able to this is gonna be pretty cool once we do it uh there's only two notebooks so far so we're gonna be the first one really wait we're not gonna be a first one we're gonna be the third one I don't know why I said we're gonna be the first one we're clearly the Third uh hand emoji let's let's get a hand wait what is the this hand mean is that like okay let's copy this into here sign language recognition Eda and then we're going to meet oh it let me have that many characters usually it tells me that that's too many all right we're gonna do this in a notebook all right so [Music] um the uh sign language Rick ignition challenge the goal of this competition to classify isolated American sign language signs we are provided so let's see what we're provided uh Landmark data from law Landmark data from Raw videos with the me media pipe holistic mod model and are asked to predict the sign in uh from this data okay so let's go ahead and do that see it people are doing this in chat what are you going to do today we're going to check out this competition uh kaggle data draft session is starting while the draft session is starting let's go to our overview look at this video we could watch this is cool I would suggest watching this I'm going to watch it afterwards I'm going to put it in chat if you guys want to watch it um but what I want to look at while this is loading up is the evaluation there's my data tab here on the right did they change this oh it's a little thing down here now they made it just like an itsy bitsy thing down here on the bottom right I found it though uh this competition you're submitting a tensorflow lite model file the model must take one or more Landmark frames as an input in return a float Vector so the thing about this is sign language can have motion right it can have motion to what your to what you're saying so you can't just take one moment in time and try to predict using that or you could but it's not going to be as good as if you have a time Dimension to it so that's why they're saying You must take one or more Landmark frames as an input and return the float Vector the predictive probabilities of each sign class oh so this is just a multi-class so we're going to figure out what all the signs are we're trying to predict the model must be packaged in a submission.zip file incompatible with tensorflow Lite runtime you're welcome to train your model using framework you want and convert it each video is loaded with the following function load relevant data subset this is a parquet path so it only reads in the X Y and Z Columns of the path data and we're going to do this evaluation inference is performed roughly as it follows as follows ignoring details like how we manage multiple videos that I'm not sure what they're talking about is this what it's going to look like interpreter.getsignaturellist.keys model path interpreter required signature not found in found signatures raise kernel eval exception require input signature is not found oh so we're just submitting the zip file of our submission.zip it's going to read that zip file in and create an interpreter out of it I don't know much about tensorflow Lite interpreters we're gonna have to figure that out and then honestly I'm not quite sure what what's going on here because it isn't going to have to run on a parquet file load relevant data prediction function gets signature running serving default okay this is where it takes the input which is the frames which is the output of this load relevant data I think well there's two months to try to figure this out oh by the way first prize is 50k that's not no joke uh and then less than nine hours to run okay so we're gonna have to figure this out and maybe the other notebook will give us an example too of How It's run but they didn't even tell us what the valuation metric is the value mission for this contest is a simple oh classification accuracy so that's that we'll put here uh data Eda let's read the CSV should I use pathlib uh so our base directory is going to be this input ASL signs train uh no that's just this baster train.csv so this is our training CSV let's see what uh what shape this is nine thousand ninety four thousand training examples uh uh train.csv has the path to each per K file the participate pent ID sequence ID and sign what signs are we trying to predict what are we even trying to predict so here's the signs we can value counts on this it's kind of big so so there's 250 unique signs uh ranging from 299 to 415 examples of each but how many overlap this I don't know so so we're going to plot the top 20. Let's uh import Seabourn and use a Seaborn color blind let's Also let's also um is the internet on accelerator no internet is on oh it's on by default so we're gonna load NB black this will make our code nice what do you mean by overlap how fast is kaggle compiler kaggles uh kaggles online free kernel slash notebooks things have changed around since I've used it last but uh they're okay they're like decent sized machines with potentially big-ish gpus um oh this is what I'm looking to click on yeah so this is you can see it has 73 gigs of space of disk space on this um I'm not sure how many CPUs it has 30 gigs of RAM they upped that and then we don't have a GPU selected but they we could potentially have a GPU why isn't this working oh yeah load extension lab black is what I want to do and pip install silent dash dash quiet I should put that here install NB black for auto formatting that's what we're doing here and then we're going to also go down here and actually load it in on the same cell as where we install it and we have our train data set so we're going to do a 10 by 8 and let's do the top 50. title equals top 50 signs all right this is why we installed MB black because when I run this cell now it's going to auto uh split these lines for us uh sort values ascending equals true maybe yeah this will give us the top 50 signs so we can see here that the top sign is the where or is the sign for listen that's exactly why they have the face right so you can do the the hand relative to the face and then let's do the bottom bottom 50 signs might as well just do them all uh zipper zipper is like the least one vacuum beside person dance man I wish dance was a little bit higher what's MB black for is for it's for auto formatting it runs uh the black black formatter on your code when you run the cell let me make sure I say that this is in the training data set is it just me or is this kind of like too big I made this too big and this should be should do it like this and then provide this the axis that will plot on and what else we'll set the x label to be number of training training examples right let's just copy this bottom and we'll do a tail here all right let's load an example of a file so let's load one of the let's load listen since listen is such a important one um let's query where sine equals listen now one thing I did want to see is are all of these gonna is there gonna be overlapping paths that also have listen Okay so uh the question I'm trying to ask here is are part is there only one sign per parquet file so let's do train path value counts on this so we'll answer our question yes so every parquet file only has one sign associated with it but what is the rest of this data is the sequence ID oh so the path is participant ID sequence ID I see now why it's formatted like that and that's exactly why it's this okay so let me undo that uh each parquet file is in the path us let's let's find this exact path to the file eat each um so now we know the path of the parquet file the parques associated sign can be found found in train.csv so we're gonna search here query um sine equals listen again there we go and we're going to pull one of these uh so we're gonna pull an example per K file data we'll do that here by doing values and pulling the first one and then we'll um what is this parquet data called LAN example landmark pd.read parquet example file name hmm no such file or directory oh that's because it has to be let's let's list all the data here wow LS is really struggling here no version information available required by I don't like this bash command all right so now we can see in this folder in this G flash lets us be colorful and gives us the backslash so we know this is a file so it needs to be on top of the base directory so we need to find this base directory basically uh bass dir and then plus the example file name and this should work example Landmark so this is the landmark we pull an example Landmark file for the sign listen let's look at these landmark let's look at this Landmark so what do we have all right so there's 543 frames no wait sorry there are uh so the file has six unique frames let's let's Loop over um let's also find how many unique Landmark types that we have uh should you check for possible data leakage I'm just going to trust that they've done a good job with that so far they also said not to identify personal identifiers from it uh like try to identify the person from the uh from these landmarks Rob what time zone is my location I am Eastern Standard by the way sorry if I'm uh late in responding to the chat I've been getting into this competition because it's so exciting what's the best compiler for data science is that kaggle kaggle's not really a compiler so we write most of our code in Python which isn't a compiled language uh hi Rob I'm so happy to watch you live I'm a great fan from France Joelle Joel thank you I hope I'm pronouncing that French enough for you sorry what are you doing what is this for this is a kaggle competition hard Rocco we're looking at a cargo competition Robert how are you getting those auto complete options as you type is that built into kaggle notebook yeah that's something that's uh built into Jupiter it's tab completion it's pretty good about it oh and um I screwed up my chat this chat thing from restream hates me so we're gonna do that all right unique type all right six unique frames and four unique types um and then we'll just do the types and videos will be example Landmark type unique and we'll just print those it's cool to have a video competition with no video new one okay so as four unique types fight face left hand pose right hand um uh let's compare for the different um for a bunch of parquet files what type of data we have so let's go like this let's go like this let's do so this is going to give us all a bunch of paths for listen listen files 4f in listen files and let's do enumerate over this then we'll do our read in and then we'll do our print and if I equals 20 will break out of this so let's see how quickly it can oh I can run through this quick all right so one thing that we're finding is that the duration of these is not consistent look at that so I have an idea let's create some metadata for the training data set so we have this might not want to run it on everything right now we have 94 000 rows and that means 94 000 parquet files we're going to import tqdm up here hey thanks burst be burst blee appreciate that how fast is kaggle compiler this isn't a compiler it's a virtual machine uh we notice the number of frames is not consistent uh almost every file has four types of uh landmarks face left hand pose and right hand at least for these examples but we're gonna do it arose one of those one of the rare times we want to actually do it a rose and then we'll do the total of this tqdm will be equals the length of train there we go so this will Loop through it and then we'll take this D and we will read in the parquet file I should really call this path I don't know like this file path let's be really descriptive with their name so it's clear what we're doing we're pulling in the file path and then we're getting an example Landmark parquet file this is cool okay so example landmark then let's make this into some metadata so this is the metadata of for this file which is the face the pose right hand and then we'll also take this frame and unique this will be frames be equal to this that's the number of frames what else do we want to get from this example landmark Landmark index well the landmark index really depends on if the face is involved yeah so we're going to be able to we should be able to plot this out hello sir Rob Melissa thanks for start live I'm your big fan from Bangladesh nice welcome from Bangladesh hi again from Ukraine great video have you work uh have to go to work and we hope no two-day big rocket so man stay safe thanks for hanging out from Ukraine that's awesome to have you in the chat all right so the metadata meta now we have this is there anything else we want to pull from this okay so the row ID is the frame the type and the landmark Index this looks similar to how I did with some other stuff um can I do AG X min max it's not like this stuff really matters why min max and then Z min Max uh Hey dope content thank you Z Sean thank you appreciate it appreciate it you don't need to donate but that's nice of you um I'm gonna gonna put it all back into the the channel in some way shape or form I plan on so how do we com add this dictionary to this dictionary so this is like our XYZ metadata min max and mean it's like our XYZ metadata now I know we could do dictionary comprehension in Python I learned about this from a YouTube video I want to say recently so dictionary from Keys range we could do dictionary comprehension like this this no I want to rename the keys how did you beautify that cell uh NB black I will send this to you right your way in the chat so you can check it out I believe the idea is to help create a model that will be then be used by apps that would help someone learn uh sign language correct correct yes this is a kaggle competition let's save this version and see if I can share it with you guys while it's going any other questions Rob do you use the work are you done working just to YouTube now oh no I work I work mostly YouTube is just a fun side Hobby this is just for fun guys I'm not doing this for the five dollar donations although I do appreciate it hi Rob do you keep these stream cods anywhere sorry I meant to ask you this so frequently ah they're all in my page so um here's my YouTube page you go on live I leave them all up here so you can see my last one's here I didn't even I was too lazy to put a thumbnail on this one and this is us live right now we've got 50 people watching nice but this will stay up here I'll try to add a thumbnail to it later and maybe crop out the beginning and then probably get dinged for copyright infringement because of the music that I'm playing even though it says it's a copyright free playlist sorry didn't mean to get angry there can I do this to rename the keys so hacky um so for each key in this XYZ meta keys I am going to create this new key which I will then append to our meta dictionary yeah new this new key will get the x y z metadata for this key X not defined oh yeah because that's this key and then another key right yes now let's look at this all together yes this is my hacky way of adding this stuff to the key so let's let's see what it looks like the metadata for this file is this and now let's do this let's do combined meta is just gonna be a dictionary where we take in what is I I is the index we'll take in the file path meta no wait not meta combined meta for the file path and then we will add this data into it if I is greater than or equal to 50 will break so we're going to try to run this on 50. so quick I mean to run all 94 000 this would only take an hour we don't want to waste an hour here but now we have our combined metadata about these files so let's do create this into a data frame that's not what we want uh oh it kind of is you just need to transform this so now we have for each of these files the number of yeah the number of rows that are for face number of rows that are for pose left hand right hand frame the number of frames X-Men yeah these values I don't know how helpful they're going to be um but let's just run this on should we be dangerous that only took one second so let's do ten thousand no ten thousand would take too long one thousand one thousand yeah it's not going to take that long uh we would resetting the index do anything yeah that would be a good idea after we're done with this um what I want to do is merge this back on to the training data frame actually there's got to be a better way I think there's a better way than doing this transpose after I load this in but you're right so resetting this index uh and it will rename Columns of index to be named what is it called on the training data set to be named path and then we'll use that to merge onto our training data set um and we'll do how equals left and we'll yeah we'll run this now we have training data with metadata we only ran it for our first 1000 of these 94 000 because we don't want to sit here and wait um but I could run this later with all of it while this is going let's failed after zero seconds why did this fail where did it fail oh it's funny that it ran it failed on their example code below code was taken from the valuation page um so let's try to run save this version again I'm going to go here into the settings of this code and make it public there's a save button here it is save changes let's go to this notebook can anyone in chat let me know if you could see this notebook give it an upvote so I know for sure toggle this up vote all right so this do you guys understand what this is does doing these kaggle competitions and always learning new things make your life at work any easier like facing challenges or daily tasks yeah [Music] they do all also helps you think about the problem in like a skeptical way because when you're doing kaggle competitions you're kind of testing the limitation of what's possible and it gives you a really good idea of what is not possible when it's brought up or what's you know what's not known to be possible yet nope so you're saying you don't see it says actively running Okay so you guys uploaded it but okay now refresh refresh your browsers and tell me if you see it now I should promote myself this notebook was created live during a live coating stream watch it on my you YouTube and twitch page YouTube is going to be how do we do URL links we do this how do I do that subscribe confirmation this is something I learned this little hacky trick if you do question mark sub confirmation equals one to the URL link then when people click it it oh wait it doesn't work maybe because it has this equal sign they click it and then it automatically will pop up for them do you want to subscribe and then when I do it says you can't subscribe to yourself you idiot but that's the idea there put this here good morning fakie very sneaky LOL yeah I learned that from someone else's links a little trick you learned it here first folks you learned it here first I'm having a lot of fun with this competition we haven't even looked at the data yet we haven't really looked at the data yet are you guys having fun put a f in chat if you're having fun F is for fun folks um g f all right so we're going to look at these columns in the metadata to try to get an idea of what are the most frequent types of land marks provided we could just sum this I need to put this in here some of rows by Landmark type so mostly face that's because the face media pipe face mesh so that's because face mesh has a lot of data points so it's going to be biased in that sense like look how many data points there are um information about the face model is in this paper man it's cool to eat this is a cool designed competition Leica we don't have to deal with video files um but that's why we have mostly face what if we do train meta and then we do this and we do Max no not Max greater than zero drop in a subset equals face this is going to drop all the columns that we don't have and then we're going to do the mean value of this I know this looks weird and it's not interesting so this is just saying in the face pose left hand and right hand are now 100 of these thousand videos that we pulled every part k files file has at least some data points for face for all four types of stuff types of land marks face pose left hand and right hand FaZe has a lot more data points because media pipe provides how many landmarks in media pipe face mesh I want to find this 468 3D data points per frame I guess I guess if it can find the whole face we're going to run this code on it in a second year uh check out one one example oh one other thing we want to do and I'm going to change this to be to run on more than just just one thousand so let's make this like n parquet to read so we don't have to load all 99k come on 94 95k check out one example and let's plot it so let's do let's load our example Landmark like this again we're gonna we're going to do a listen one what's the average frame okay so like let's take let's query where the frame equals 38 middle of the video oh oh I just realized something X Y and Z values can be null what's the context of the greater than zero um so this is checking to see if the number of landmarks for this this type is zero so what I'm realizing is that I didn't realize before is that these values can be null so before when I was creating that metadata I was just taking example landmark and I'm doing type value counts and it assuming that we have values for each of these but I should do drop n a subset like x y and z that gives us the actual amount that we have so what what happens is yeah so this plot is wrong because they provide a row for everything even if it doesn't exist then they provide null values you guys following me there you got to trust me here so we're gonna rerun on these 1000. get the number of landmarks with x y z locations uh uh data per type super fun competition already with the subset what's the subset um did I do a subset too yeah so this drop in a subset is just because we ran only on 1000 of the parquet files so otherwise this average would be messed up I'm not doing a great job exam explaining this foreign the brake condition is wrong you're right thank you I would have sat there for like two hours without if you hadn't said something maybe we should just do some media pipe stuff to show how it works but we should probably plot this data first foreign so it ran a thousand we're merging it on now we're gonna plot still we have the most of face but here's the real question yes hold on a second foreign s are null interesting so I need to do this query index is less than a thousand because this drop in a is not going to do work and then we'll do fill in a with zeros I don't believe this I must have done something wrong so we're saying that the face there's always something for the face so here we're in the middle of the video and yeah there is stuff for the face so the face is always there every time here is the percent of examples percent of frames with data percent of frame slash key points with data so we always have a hundred percent because it's a rate because it's not a percent um right hand more than left hand but these are about 50-ish percent of the time that they exist otherwise we don't have the key Point data for them heck yes data science Johnny you know what's up you know what's up all right so we want to take our train with meta example and we wanna drop an a subset of let's just drop an a this will drop any examples where we don't have all all three face pose right hand and left hand which happens to be way over here so we're going to subset to that that's going to be our example all right so none of them have listened uh what's another popular word up here or sign look look listen all right we know oh shush should be a good one all right so this frame median is 25 so this is giving us the middle of this example and let's try to plot all right three 3D uh plotly 3D plot so we're gonna do a 3D plot 3D plot of landmarks from example and we're going to do import plotly Express as PX import plotly Express as PX PX three line 3D no not we want a 3D scatter plot scatter 3D uh our data is gonna be this example frame that's a single frame come on auto shift tab used to show me the doc string they've changed it why have they changed it X Y and Z is going to be all right so this is going to be pretty simple data is example frame X is going to be X y's y and Z is z we're gonna color by it is it called color what we call it out color is by the type hopefully this looks cool what in the world so this is the person's face apparently maybe they're not in the same dark thing off let's turn on face all right there's the person's face do you see it do you see their face some signs not going to have every column what do you mean by that five shot don't you need to have data for each column not sure what that means so this is someone's face there's their face but why don't we see the hand maybe because yeah there is no right hand but I thought we picked one that dropped n a right did we run this it could be that frame so Group by frame X is an a no x y z is going to be this and we're gonna Group by the frame no x y z is some so this means that frame okay so this is what's going on at frame 19 and 17 . uh so this is the number of missing missing values per frame and it looks like 17 and 19. pick frame 17 because we have no missing XYZ data all right so let's go here delete this and do 17. there we go it's the 3D space is is kind of weird oh I wonder if we could like Orient this in a way that makes a little bit more sense but let's try to just show the left hand it's gonna be hard it's going to be hard to plot this are they relative let's see left hand to right hand right hand left hand they're over each other they're over each other all right so um where's my music where's my music cat man I wish I could work on this one for like days Days on days are you guys having fun hey Rob if you can do a session for data analysis using pi spark data frame I have a YouTube video about that but no specific session yet that's a good idea let's save this version let's also save what I'm going to do at the end of the night is I'm going to take train with meta and I'm going to say 2 per K train with meta dot parquet at the end of the night to be super crazy I'm going to remove this uh or make this n parquets to read larger and then I'll just run this notebook and then we'll have that data as the output after an hour of it running that's what we should have um so this is kind of hard to see what I want to see is with media pipe plot landmarks and some has someone already done this they just went straight to training a model I don't know what this is this isn't the data set that we okay so they read in the new parquet file then what is this sign to prediction index map okay so they're just encoding them to numeric values then they're reading in sign language mnist yeah this is just this is just um just going to confuse people because probably a tutorial they took from something else right might be missing it but this isn't the data we're given in this competition it's just another mnist hand um data set so let's go look at other people's stuff read one parquet file they're using dask data frames okay they're small files so using desk might not be necessary what we want to do is actually plot these using media pipe um they said they used holistic what did they use to create this data these notebooks were just a few hours in so not that many notebooks to look at look at this overview data oh okay so I was missing this I was missing this that's what he was pulling in here this makes sense now that these are mapped to numeric values sequence ID XYZ hand Landmark locations can be found here but we want all the landmarks media pipe holistic model so it's this one so this is how we draw with media pipe holistic for static images the 3D plot is kind of hard to see um try to use media pipe to plot so this is actually using media pipe holistic model to read in a file reading a bunch of files actually print the coordinates and then this is what we want to do is draw on on this XYZ coordinate right I think that's what we want to do so let's try to do this on an image let's try to do this on a whole person example uh like they show here holistic why do I keep on putting W in for holistic yeah where's this guy media pipe holistic okay I want that image I want this guy um let's try to do this this image maybe like this all right this image let's see if we can run it on weird weird search query but okay I was just trying to get my hands uh all right so we'll get this uh pull an example image run media pipe holistic to see how it produces the results plot them on the image so this is pretty easy because image files is going to be this image uh no module name media pipe seriously we're gonna have to load media pipe quietly shush hello I'm new to DS excited to learn new things you're in the right place King kid King cold exclamation point YouTube we'll take you to my YouTube channel or youtube.com LOL Panda spark yeah oh shoot I don't know if it installed so correctly but let's try it that's the pose estimation no no no no no no how is that the pose estimation of this image files there I am drawing landmarks pose face masks face mesh so let's see what the face mesh landmarks look like for this I'm basically trying to to hack it so I can see what the landmarks look like so then I can put the data from this into that format to try to view it but why is it not showing the beautiful plot of the nose coordinates let's pick a better image how about that our Y and Z I think it's just getting messed up because there's not a full person in the picture person for pose estimation example single person um so I don't want the pose estimation to be drawn on them already person standing basketball no person standing here we go so all photo it's all uh watermarked but we're gonna not do my picture anymore instead do this person's picture full length fashionable young man standing on isolated white background now that's a detailed name whoo maybe I need a I am read this image show this there we go there we go this is what we were trying to show um so this is the isolated image of this person let's also uh let's get mine now I wanna I wanna um rub my hands hands there we go we got hands in this no I don't want that I want the good things so now we have two images we'll process that's HQ default and the this full length portrait of a fashionable young man standing on an isolated white background I don't know why it plots this this throws me off all right so now we'll do LS in our temp directory oh this is um annotated image star will be yeah so we have zero and one so we'll do zero and one look it got my thing but it didn't get my hand what the it cut out my hand and we're in the same t-shirt but now we get an idea of what this looks like now let's reverse engineer to try to I don't know if this is going against their rules about not trying to personally identify someone I hope it's not against the rules try to use the same format for plotting on new of parquet data so clearly we won't have this background image like the the person's face but we could try to plot with these colors and stuff what the hand looks like so I wonder what the size okay so this is where our media let's try to get the bounds train with meta X min x max just trying to get the an idea of the image size that we want to make so the annotated image is just an array of shape like this so let's make background image as numpy zeros like this shape background image so if we uh zeros like this oh just zeros I think yeah now we have this is a numpy array of 3 000 by 825 this is the dimensions of the thing you guys following me is this fun someone say something in chat I'm having fun so this is just gonna show like a black image that's not