AI Transforms Google Photos into Real-Life Scenes

Key Takeaways

using tourist public photos from the internet they were able to reconstruct multiple viewpoints of a scene conserving the realistic shadows and lightings this is a huge advancement of the state-of-the-art techniques for photorealistic scene rendering and their results are simply amazing let's see how they achieve that and some more examples [Music] this is what's ai and i share artificial intelligence news every week if you are new to the channel and want to stay up to date please consider subscribing to not miss any further news researchers at cornell university introduced a new way to use online public photos taken by tourists to construct a continuous set of light fields and synthesize novel views capturing all times of day scene appearance the complexity behind this task is that all the pictures are taken at different times of the day different seasons and different orientations in order to answer this problem they introduced deep mpi which is a new multiplane image representation that does exactly what they needed their method is completely unsupervised needing zero information other than the photo itself from the internet and allows real-time synthesis of photorealistic views that are continuous in both space and lighting you can see how much better the results are compared with the previous state-of-the-art models now that we've covered what they've done and why it's so impressive let's see how they have achieved that and some more results in short they synthesize arbitrary views of a scene with continuous viewing condition such as lighting by using pictures from the internet of multiple lighting and angle sources it takes unstructured internet pictures of a specific place and learns how to reconstruct a representation of the light field that respects the real world shadow physics as you just saw previous works like fields are inconsistent through the scene which is the greatest contribution of the paper this is done with a two-stage models architecture at first they use their new deep mpi representation they start by reprojecting every image to the reference viewpoint and averaging all these reprojected images at each depth plane thus creating a mean rgb plane sweep volume psv which is a set of views wrapped with disparities in a given range since this mean rgb psv cannot accurately model a scene content that is obstructed in a reference view they introduce the second phase of their network the second part optimizes the latin features in their deep mpi representation using an encoder and a rendering network it is able to capture and re-render time-varying appearance the encoder's role is to produce an appearance vector from an exemplar image and an auxiliary deep buffer containing semantic and depth information of the scene the deep buffer allows the encoder to learn complex appearance by aligning the illumination information in the exemplar image using the scene intrinsic properties encoded in the deep mpi representation without this alignment the results will be as inconsistent as the previous work we've seen this align deep buffer is the main reason for the realistic shadows and lightings in the rendered scenes then the rendering network represented by g in this model's architecture takes both the deep mpi projected to a specific target viewpoint and its appearance vector produced from the encoder and predicts the corresponding rgb color layers this rendering network is a variant of a u-net architecture with an encoder decoder architecture called aiden used for style transfer applications this model produces natural scene appearance while stabilizing the training preserving the color and style of the exemplar images i linked the aiden's architecture paper in the description for more information in short given a specific exemplar photo they were able to synthesize novel views close to the reference point while preserving the exemplars appearance it is mind-blowingly accurate just take a minute to see these results with multiple lightings the link of the project website is in the description with the code and data set coming soon as per the authors do of course this was just a simple overview of this new paper i strongly recommend to read the paper linked in the description for more information please leave a like if you went this far in the video and since there are over 90 percent of you guys watching that are not subscribed yet consider subscribing to the channel to not miss any further news clearly explained if you would like to start or improve with machine learning i've linked all the best online courses in a reporter in the description thank you for watching [Music] you