FoundationStereo: A Foundation Model for Zero-Shot Stereo Depth Estimation | NVIDIA Research

Key Takeaways

we present Foundation stereo a foundation model for zero shot stereo depth estimation notice the sharp reconstruction details for the key hook which is Tiny and includes translucent hanger and metal pieces our method produces sharp dense and metric scale depth given the input of stereo images here's the outline for the rest of the video all results are obtained by zero shot inference of the same model weights and parameters without any fine- tuning now we show results across various domains in the wild here we see an example of a cluttered desk given a pair of stereo images show on the top left our method generates accurate cylindrical Point Cloud for the disinfect bottle sharp and perpendicular edges for the napkin box the quality maintains for more distant objects like a wallet and a package box looking from Top the white cylinder water bottle on the left is accurately Recon constructed despite its textureless and partial visibility here is the room of robotic arm the arm consists of metallic links purely black gripper the table is also purely black often causing challenges for depth sensing other challenges also include the cube of small size as we zoom into the the background objects the trapezoid box with black hole is Faithfully captured so as the trash can with smooth curvature also look at that K with thin spout and those shelves with thin vertical structures here is a kitchen scene captured from egocentric view the tap and sink features common challenges of metallic textureless surface and thin structure the shape of the tap has decent curvature in the resulting Point Cloud looking from Top the textureless wall has nearly perfect planer surface and perpendicular Corner in the point Cloud here is a more different Warehouse scenario the original image is captured in a complex lighting with strong reflection on the ground and extreme exposure of far backgrounds under sealing light the box and shelves are straight the ground is almost perfectly planer the gray and green brackets are sharp as we look over from the high ceiling the entire structure of the warehouse is easily noticeable from the point Cloud this includes a pallet which is already far away from the camera appearing small in the image moving on to outdoor scenarios here is a front view of a house captured under an unusual perspective beside a car with shadows and sunlight the front staircases of red brick with curvature is Faithfully reflected in the point Cloud the sidew wall and garage door have planer surfaces zooming into the front door look at the small white plant base which is also already far away another outdoor scene of a street with much further depth distribution as we rotate to the lateral side the u-shaped rack is clearly visible even though it is thin as we look over the car across the street has the straight reconstruction in the point Cloud the far distance tree is vertically standing next We compare with state-of-the-art stereo matching methods all methods are run in zero shot the best best performing weights of each public released model is used to reflect the best possible generalization results in the wild in the disparity comparison methods are challenged by the reflective floor microwave ours is also the only one that captures the hole of trash bin in the Turning corner now let's visualize the point Cloud comparison comparison methods KOCO V2 CR stereo igv fails to identify the ground plane zooming in the microwave inset within the dark cabinet as well as the reflective griller introduced challenges the ceiling and floor are almost perfectly straight in our predicted Point Cloud the Turning corner with vertical walls is sharply visible here is another example involving challenge of repetitive texture on the wall which usually cause issues for stereo matching the point Cloud predicted by our method recovers the straight walls and its perpendicular connection with the ground this is consistent under different viewing perspectives next we Show an example on a cluttered robotic manipulation scenario at kitchen from the disparity Maps comparison methods tend to struggle with reflective and textureless robotic arm cables microwaves scenes Behind the Chair and sealing light now we visualize the point Cloud although it was not easily observable in the disparity Heat map our method also yields straight countertop most comparison methods have the background connected with the ceiling light moving on to the outdoor Street scenarios our method predicts correctly the translucent store Windows car windshield in the point Cloud our method yields the most accurate and complete Caro Cloud despite the challenge of strong reflection and sunlight exposure looking further the white canvas roof with smooth curvature is Faithfully recovered though it's hardly visible even in the input image finally we compare with state-of-the-art monocular depth estimation methods including depth anything V2 and depth Pro both methods have the metric scale version while they produce decent depth maps there exists noticeable scale difference from our stereo Point Cloud here all Point clouds are aligned in the same left camera coordinate frame with origin being the optical Center the walls from comparison methods are also less straight and non-perpendicular at the corner now we visualize detailed Point Cloud the tap and hand Parts become warped in depth anything in depth Pro in contrast they appear to be sharp in our results when looking over again the wall and depth anything V2 and depth Pro are less straight and non-perpendicular at the corner as opposed to ours demonstrating our accuracy their inaccurate scale prediction also results in misalignment with our Point Cloud thank you for watching