Build location-based augmented reality with ARCore geospatial API

Key Takeaways

Welcome to our Tech Talk. I'm Belal Sabdu, a product manager, and I'm joined today by Simon Lynon, a software engineer, and we're from the Google Maps team. Today, we're really excited to introduce you all to the ARCore Geospatial API, which will allow you to build global scale immersive location-based AR experiences on Android and iOS. We've been mapping the world in immaculate detail and scale over the past 15 years using satellites, aerial, and ground-level imagery to build an unprecedented understanding of the dynamic physical world. In totality, we've covered more than 10 million miles of Street View, a distance that can encircle the globe more than 400 times. We've also covered more than 36 million square miles of HD satellite imagery, covering more than 98% of the entire population. From this massive corpus of billions of images, we have fused them together into a 3D world model that geometrically and semantically understands the world, and it is the underpinning foundation for our first-party applications like Google Maps and Earth. This 3D world model is optimized for visualization, analysis, and localization. In other words, maps for humans and machines alike. Today, we are here to talk to you about the machine readable 3D model, and in particular, the smartphone camera in your pocket, and how we're enriching it with Google's understanding of the world to allow you, the developer community, to create immersive world-scale AR experiences. We're excited to partner with the Google AR teams to bring this 3D understanding of the world to you through Google's AR developer platform, ARCore, which allows Android and iOS developers to build immersive AR experiences for mobile. This technology is called global localization and has been stress tested in production powering the AR live view mode inside of Google Maps and the AR places filter in Google Lens. And today, we're excited to open up the same core capability to you, our developer community, to create immersive location-based AR experiences in your own applications. This capability is available in over 93 countries across the globe everywhere we have Street View. With that background in mind, let's get right into it. We'll start off by discussing different options for building location-based AR experiences. Then, we'll provide background on the technical capabilities on our road map, as well as walk you through the fundamentals of using this API and just how easy it is to get started. And finally, we'll share what our early partners are already doing with the geospatial API to address a range of verticals and use cases. Let's take a step back. If you want to build location-based AR experiences today, you have a couple of options. Use built-in sensors like the GPS and compass on your phone. Use spatial anchors like the ARCore Cloud Anchors API. And finally, use the geospatial API, which is the subject of today's talk. Let's understand the benefits and tradeoffs of using these various options. You've got GPS, which is a global scale system, great for building location-based experiences. But, if you've ever dealt with your blue dot jumping around, especially in an urban canyon, you know that the accuracy is a far cry from what you need to build AR experiences that reliably augment the camera view. Even after you do that crazy eight pattern for calibrating your compass, with location services, you'll typically get 5 to 10 m of positional accuracy and 30 to 45 degrees of rotational accuracy. This means that your camera may not even be looking in the right direction or your AR elements are completely out of frame. You also have the ability to use spatial anchoring capabilities like ARCore Cloud Anchors. In this case, you need to explicitly map a space or place of interest to attach content to it, and then can reliably localize against it. Cloud Anchors works really well at near field distances for private spaces or smaller event scale experiences, where you can physically show up and map the space before attaching AR content. But, with larger scale deployments of Cloud Anchors, discovering content is a challenge, and so is placement. We've heard from you, the developer community, that you want to guide users to the right location, to be able to remotely place content without physically going to those locations. Both of which are hard to do with Cloud Anchors today. Enter Visual Positioning System and Global Localization. Thanks to Google's decade plus of ground level imagery, we've already mapped the world in immaculate detail. By working in conjunction with location services, we can take a good GPS estimate and make it great, ready for AR. So, with the ARCore Geospatial API, you are able to remotely attach content to the world and guide users to it, allowing you to create immersive location based AR experiences almost everywhere we have Street View. All right, let's summarize the differences. With GPS, you get global scale with lower accuracy and variable reliability. With the Cloud Anchors API, you get high accuracy with an unlimited scale because you explicitly need to map a space. And with the Geospatial API, you get the best of both worlds. Remotely placing content in the world and offering nearly ubiquitous accurate localization and heading without needing to manually map the space. We hope to surface additional capabilities over time to help you build truly next-gen experiences. I will now hand it over to Simon to unpack the underlying technologies powering the Geospatial API and the capabilities it gives you to turn the world into your canvas. Thanks, Bilawal. In the following minutes, I walk you through the three pillars of the Geospatial API. Geo localization, geometry and anchoring, as well as world understanding. Localization is the foundation of the Geospatial API. It is key to bring your AR experiences to the global scale by allowing you to place virtual content in relation to the globe. To provide this capability, the device makes a server call to the Visual Positioning Service, VPS, where your geographic coordinates and images are used to match against high-resolution 3D data from Google Maps. This process is called visual localization, and it precisely locates your device with respect to the environment much more accurately than what was previously possible with GPS alone. I want to take a moment to run through how all this works. Street View images, which have been captured around the globe for more than 15 years, are the foundation of VPS. Deep neural networks identify and describe those parts of the images that are likely to be recognizable over long periods of time. You can see that the algorithm has learned to focus on buildings instead of cars. We then combine these salient points across tens of billions of images to compute a 3D point cloud of the environment. This localization model consists of trillions of points and spans nearly all countries where Street View coverage. When your device makes a request to the Geospatial API, a similar process is applied to the image. A neural network processes the pixels to find recognizable parts of the environment and matches them to the localization model. Computer vision algorithms then compute the position and orientation of the device that is sent back to you. This process makes use of TPUs, Google's machine learning accelerators, and it takes less than a second. Using the response from VPS, the Geospatial API takes care of merging your local coordinates with the geographic coordinates so you can work in a single coordinate system. As Belovod mentioned earlier, this launch offers a very rare combination. The Geospatial API builds on the Visual Positioning Service, which is in a very mature state, and we make the API available for nearly all areas covered by Google Street View, which is very widely scaled. One key challenge when building AR experiences is the accurate placement of content in relation to real-world geometry. The API we launched today allows for anchor placement at a given latitude, longitude, and altitude. We expect to launch terrain anchors later this year, which will allow you to create anchors by providing only latitude and longitude information, while other information from Google Maps is leveraged to precisely place the content at the right altitude above ground. The image on the right shows an example where the precise altitude information is key to a delightful user experience. In the following, I would like to give you a behind-the-scenes look at a technology that the team is currently working on. These capabilities are not launching today, but we're exploring ways to make them available for developers in the future. AR experiences become truly magical when they seamlessly integrate with the real world, for instance, through occlusions and physics interaction. We believe that one key enabler is given by abstract building geometry as shown in the image on the right. Another angle we are exploring is to bring already familiar AR concepts to a global scale. For example, ARCore provides the ability to anchor elements to physical locations and snap them to plane geometry detected by the device. For the geospatial API, we're thinking about a similar concept which we call geometric anchors. It consists of two parts. The first part is to capture and persist intent such as keeping anchors that were attached to a wall sticky to the wall. The second part is about maintaining anchor poses. So that even if the our models are refined over time, anchors remain at the location that you intended them to be at. Lastly, we will hope to gradually expose more of our underlying 3D semantic model which will allow you to build even richer experiences. In particular, in combination with existing geospatial offerings such as Google Maps platforms. Adding semantic understanding to the geospatial API will allow you to procedurally place content at places in the world such as a specific fountain in a park or your favorite coffee store. We're looking into making pieces of the semantic 3D model available in the coming quarters. After this exploration of the future, let's take a quick moment to look at what we are launching today. This code example shows you how easy it is to bring your AR experiences to a global scale. To obtain geospatial data, you need to enable the geospatial API by changing the geospatial mode in your session configuration to enabled. To determine an anchor location for remote anchor creation, you can for instance sample points from Google Earth or Google Maps. You can then start creating anchors in the global frame using these coordinates. Another example I want to show is how you obtain a geospatial pose in your app. A geospatial pose consists of latitude, longitude, and altitude information, as well as device orientation. All these quantities are provided with estimated uncertainties, so you can adjust your experience accordingly. You see that only a few lines of additional code are needed to build global scale AR experiences with the new geospatial API. And with that, I'm handing back to Bilawal to walk you through some use cases. Thanks, Simon. We've seen developer interest for a range of use cases and verticals. Today, we'll focus on three sets of applications. The first vertical is ride sharing and micro mobility. Both demand high accuracy location for a range of use cases where GPS just doesn't cut it. Thanks to our new API, partners are removing the friction from parking dockless e-scooters and e-bikes, adding pinpoint accuracy so that riders know exactly when their vehicle is in a valid parking spot. The result is greater peace of mind for riders and more properly parked scooters on city streets, leading to better relationships with the communities that our partners serve. Conversely, vehicle discovery gets easier, especially in dense urban corridors, with the increased location accuracy, which our partners are using to help customers find vehicles, thus improving their user experience and increasing vehicle rental rates. The next vertical is location-based AR experiences. Thanks to our API, developers can focus on creating compelling user experiences that provide utility and delight, without needing to build and maintain maps of multiple locations. Telstra and Accenture are creating a companion app for Marvel Stadium that brings customers to their seats while showcasing engaging immersive content along the way. Docomo and Curiosity are turning Tokyo hotspots into an immersive world where you can fend off virtual dragons with your favorite robot companion. Finally, we have gaming and self-expression. With the geospatial API, you can literally turn the globe into a canvas for creativity. In Balloon Pop, you can place balloons as targets in the real world and try to pop them using a physics-enabled ball that reacts to distance in world space. Nearby users can join the experience and pop the same balloons via a simple Firebase implementation. And with Pocket Garden, you can adorn your neighborhood with a colorful AR community garden, accurately placing seeds across the real world at scale, planting vines, puffy succulents, and more. We've also given Snap early access to this API to help explore entertaining and educational use cases. All of these are great examples of what is possible when you turn the globe into a canvas, and we're excited to share that Balloon Pop and Pocket Garden will be open sourced and available to you to bootstrap your own creativity. We look forward to seeing what developers build with the ARCore Geospatial API. So, to summarize, with the introduction of the geospatial API, we're providing the foundation for building global-scale AR experiences. The API is mature and production ready with the underlying technology deployed at Google scale for more than 3 years, and with a range of partners using it in their own experiences. The geospatial API is available today on all supported ARCore-enabled devices across Android and iOS. Get started with our new geospatial code lab or check out our samples for Android, Unity, and iOS in the ARCore SDK. We're excited to see what you create when the world is your canvas.