Phong Surface: Efficient 3D Model Fitting using Lifted Optimization

Key Takeaways

we present the phone surface model which enables efficient 3d motor fitting using lifted optimization our guiding example of motor fitting is tracking the 28 degrees of freedom of the human hand by fitting to smash 3d data doing that in a head-mounted augmented reality system such as the hololens 2 enables a natural interaction mechanism unlike anything we use in today's computing systems [Music] on the hololens 2 there is a powerful cpu and gpu but they are reserved for applications so hand tracking must be completed in just four gigaflops on a digital signal processor that's about one percent of the processing power over iphone 7. the k2 efficiency in the previous work was the use of a smooth surface model subdivision surfaces or base plants the smooth surfaces allowed the use of a method called lifted optimization which greatly reduced the number of iterations used in motivating and allows the use of many fewer data points although individual service evaluations were 7 times more expensive compared to a polygon surface the overall reduction was a win but to get real time in four gigaflops we needed to get that 7x back to do so we introduced a new surface model the phone surface it's nearly as cheap to evaluate as a polygon mesh but preserves the benefits of limited optimization that is fast convergence with fewer data points the inspiration for the phone surface comes from the fun shading technique in computer graphics the idea is to use the polyhedral surface model but to interpolate the surface normals let's look at this in detail on a 2d example we take the non-smooth polygon model and interpolate normals as if from the smooth surface now let's see what happens when we try to fit the model to some data we form correspondences as usual from data to model and as it's common we don't just match the closest point but we minimize a weighted loss combining distance to the surface and agreement of the surface normal with the data notice the blue arrows these indicate the contribution of the surface normals to the correspondence updates in lifted optimization the polygon model does not have these contributions because its surface normals are identical at any position within each phase these updates improves both speed and accuracy here we show a comparison of fitting methods for two toy examples a kidney beam and an ellipsoid we compare three types of surfaces subdivine surface foam surface and triangular mesh and two types of optimizations lifted optimization on the top row an icp or iterative closed point on the bottom row in all cases lifted phone converges as fast as lifted subdiv but has the wrong time cost of just triangular mesh on the harder kidney bean shape the triangle mesh is not only slower but the lack of pull from the surface normals means it has more local minima taking this back to hololens lifting means we can use just a tiny subset of the data the green dots shown here and the phone surface means we can do so cheaply and reliably putting it all together we have fully articulated tracking which deals with both hands jointly running in real time in four gigaflops of compute our technique applies not just to hand tracking but to any situation where efficient surface motor fitting is required particularly on low power devices thank you for watching you