3D Pose Estimation with ml5.js

so I see some blue maybe some pink maybe if we Zoom all the way in there can we see somebody doing jumping [Music] jacks all right so this is an addendum to another video that I made that looked at the body pose model in ml 5js for keypoint estimation but the only thing I focused on in that video were the 2D key points how do you find the XY position of various points on the body using move net and another model called Blaze pose well the blaze pose model it turns out also can estimate a z position and what that zv value is and how you can work with it that's what I want to investigate right now I'm going to start with the code exactly as I left it in the previous video I have a variable to hold on to the body pose model that I'm loading in preload I also have a variable for the video which in this case is coming from the webcam right here I call detect start to begin the process of feeding the frames from the video into the movet model whenever it receives results those results are stored in a global variable called poses and then in the draw Loop I'm drawing pink dots for all of the key points as long as the confidence threshold is greater than point1 and blue lines to connect the key points according to the body POS model skeleton connections I've also buried at the bottom a mouse press function just to console log the results from the model in case I need to examine them and we can see there's a key points array with 17 points and named Parts each with their own XY and confidence score notice that the parts only have an X and Y the key points only have an X and A Y I'm here to look at what happens when I change from the movet model to blaze post the first thing you'll notice is there seem to be more points we can see that reflected here there are 33 key points let's dig into some of these properties key points is exactly what it was before there's an XY a name for each body part as well as a confidence score if I go to key Point 3D you'll see aha suddenly there is an x y and z now those numbers look very different than the numbers that we saw under the 2D points these are pixel values what are these values they are in fact realworld measurements those are measurements in meters The Blaze pose model takes all of the key points of the body and places them within a 2X 2x 2 meter box now one thing you have to realize is that this XYZ value is not some kind of distance from the camera it's actually a relative position to the center of the body and in fact Blaze pose considers the center of the body as the hips this will be easier I think to dissect and unpack when I have the example running so I'll return back to it but I just wanted to give you a taste of what the values look like I also want to point out that under each named property there are both the XY pixel values as well as a separate key Point 3D property that has the XYZ values notice how the estimated 2D position and the 3D position each have their own confidence value they're probably going to be similar I would guess but maybe for certain images the model is more confident about its 2D position estimation versus the 3D one now if you watched my previous video I had this very awkward setup where in order to get the full body uh viewable by the camera I got a second camera I put it over here I wandered in the corner I got on a step stool it was kind of a bit of a mess so I'd like to try something different and in fact this could be helpful to you certainly it's good while your coding to get up and move around and stretch but if I'm trying to debug how this model Works make changes creatively explore the possibilities of visualizing the key points that are estimated it could be really helpful if I just had a recorded video playing in a loop so that's what I'm going to do right now and in fact I've already done that and I uploaded two video files to the web editor here Dan jumping jack and Dan 3D test so to start let's just replace the live camera feed from one of those video files I'm going to leave the variable name as video I'm going to change from create capture to create video then I'm going to add the file name of the video as a string then with a video file I need to call doplay and there we go but I'm going to change it to Loop so that it's looping over and over again other thing I'm going to do with this example is not draw the video in the canvas itself this is because I'm going to set up the canvas ultimately as a webgl 3D scene where I can rotate around the key points that I'm rendering that body's skeleton so let's take out video. hiide the video is actually uh 640 by 360 so I can change the canvas size and then I'm going to take out drawing the video and just put in a black background let me change the size of the dot so that I could see them better where they're clustered so close to each other okay there we go now there's all sorts of things I could try to do I could leave the drawing in 2D but maybe use the Z value to affect the color or the size of one of the dots but I'm going to try to actually render this in a 3D scene using the P5 web gel renderer to do that I just add one more argument to create canvas immediately you'll notice that the body is now in the wrong place this is because in web gel the origin of the canvas is the Center not the top left this is actually a good thing for what I want to do because I'm going to use the XYZ values which if you remember are relative to the center of the body which is 0 0 all I need to do is swap out the key points array for the key points 3D array in my code I can do that right here okay we're getting somewhere I see a dot in the center but the lines are still off to the right and Below let's use the 3D array for the lines as well and then instead of drawing the line with the line function I'm going to use begin shape end shape and vertices which I think will make things a little bit uh easier to work with if you've never worked with begin shape and end shape before I'll refer you to my video about custom shapes in p5js so I see some blue maybe some pink maybe if we Zoom all the way in there can we see some body doing jumping jacks look at those values remember those values are in real world units meters they're not pixel units so in order for me to see them on a canvas that's 640x 360 I need to expand the range one option would be to scale the values just by multiplying them by some number for example I could say key point x * 100 y * 100 oh and I never put the Z in here so let's put that as well and there we go there's a little jumping jack man probably would make sense for me to put that number 100 in a variable and always be multiplying everything by it but there's another way I could actually use the scale function which would in effect scale the range of the canvas itself so right at the beginning of draw I could just say scale 100 let's scale things up a bit more if the range of the values from Blaze pose is between negative one and one and I want the top to be negative 1 and the bottom to be one I could scale by uh the height of the canvas divided by two now the dots don't seem to be visible anymore maybe they're just too small me make a stroke weight of 16 oh and interestingly enough they're kind of behind the lines this has to do with the way things are being rendered are no longer a 2d canvas of layers but a 3D scene where there is depth sorting based on where the different shapes are I'm not going to worry about that right now um as I continue working on this maybe we can investigate different visual ways of rendering but immediately don't you notice something the hips are locked into place so while I'm actually in the video jumping up and down and when we used movet and the 2D key points we saw that motion go along with it now everything is moving just only relative to those hips let's see if we can make this look a little bit more like a 3D scene by adding a floor so I'm going to draw a square at 0 0 that is uh wi divid two and I'm going to say rect mode Center and give it a transparent fill where is it oh oh I'm losing my mind here I forgot about the scale I'm like why is it so big well it's scaled as well so I want that to maybe just be one meter wide terms of the units that I'm working with there we go then I want to rotate it along the x axis so that it's flat 90° then I want to move it down to where my feet are presumably so let me translate down a meter I'll make it 2 meters wide not the most amazing 3D scene work but I wanted to put something there just so you could see let's now spin the scene round on the Y AIS so I'm going to do that at the beginning of draw right after I scale rotate y by some angle I'm going to make that angle a global variable equal to zero and then I will increase that angle a little bit every frame whoa that's pretty spinning too fast aha now we can really see those points are living in a threedimensional space it's most pronounced by by the feet uh where the toe and the ankle are obviously a different place along the z-axis but luckily or unluckily for you I'm not sure which one I recorded another video where I tried to move a lot more towards the camera that one is called Dan 3D test so I'll let you be the judge of how good a job the model is doing it accurately estimating the 3D positions of the relative parts of my body in these very awkward uh poses but you can see I'm getting real 3D data there by using scale um by drawing with the webg render in P5 I'm able to reconstruct the body skeleton in a virtual space one thing that might be helpful here is not to actually do my own rotation around the Y AIS but to instead call a special function in P5 called Orbit Control and what Orbit Control allows you to do is click and drag the mouse to change your view of the scene so I can do this and I'm actually looking at it now from above so I can pinch and zoom to kind of go in closer as well you can see there's a lot of noise and jitteriness of the data probably adding some smoothing uh to the code with lurp could be something to try I'll try to include that in the code examples that I provide with this video another thing that I should mention is I'm drawing the lines and the points at their 3D positions in space but it often makes sense uh depending on how what kinds of things you want to render to actually translate to those positions and draw the shapes relative to that so let me just show you what that would look like in other words instead of drawing the point at XYZ I would translate and once I've translated I can draw the point at 0000 0 of course I don't see them any anymore that's because calls to translate are cumulative and in order for each translation to be Standalone I need to use push and pop this is also covered in my Transformations video in p5js the reason why I'm doing this is maybe instead of drawing a point I want to draw a box and drawing other kinds of 3D shapes require translate so I could draw a box of size uh 0.1 probably and let me change the stroke equ to one maybe give it a little transparent fill and now each of those boxes could potentially rotate let's rotate them along the along the z-axis let's put the angle increasing back in and you can see now all of those boxes are rotating so I didn't really have a goal here uh of something to make I just wanted to show you that with the blaze pose model there are in addition to the XY key points uh XYZ key points I wanted to talk about what the uh units of measurement are with them and how you work with them in webgl so I don't know hopefully you can come up with some creative ideas of how you might want to render in 3D or use the keypoint Z estimation in some other creative way please share those things with me you can do so on the coding Train website the page for this video with all the code examples is linked below and I'll see you next time on the coding train [Music]