Introducing fVDB: Deep Learning Framework for Generative Physical AI with Spatial Intelligence

Key Takeaways

[Music] the real world is large and we want to create digital twins at the scale of reality to do this we need to build algorithms which scale to the size of reality algorithms like Citys scale neuro Radiance fields and algorithms like mesh Reconstruction from a billion lar points the main bottleneck to building these algorithms is deep learning infrastructure we need infrastructure that handles large spatial scale to capture the size of the world we need infrastructure that can handle High resolutions to simulate the fine details of the world and we need infrastructure which can be distributed on multiple gpus to leverage computation at scale to address these infrastructure needs we've developed fvdb a deep learning framework for sparse large scale and high performance spatial intelligence fvb builds AI operators such as convolution rate tracing and attention on top of VDB the best-in-class acceleration data structure widely used for large scale and high resolution graphics and simulation building AI operators on VDB enables building spatial Ai architectures and algorithms that scale to the size of reality thus fvb is the missing infrastructure for 3D deep learning at large scales and high resolutions we now show several applications illustrating how fpb can be used in practice given an input Point Cloud we can reconstruct a triangle mesh using a sparse convolutional Network built with fbdb we can also use fbb as an acceleration structure for rendering enabling us to train a neural Radiance field at the scale of the city here we built a sparse hierarchical diffusion model using fbb capable of generating geometry at the scale of city blocks the city shown in this video is fully AI generated in this example we use a neural network built with FDB to upsample simulations on the left we see a course simulation that runs in real time the network trained using fvb upsamples the core simulation to a high-res simulation on the right we presented fvdb a framework which unifies best-in-class Graphics technology with AI to unlock spatial intelligence at large scales and high resolutions FDB is built from the ground up on core Nvidia Tech such as Nano VDB cutless tensor cores and Cuda fvdb is implemented as an extension to pytorch allowing it to be combined with other libraries such as warp for simulation to build spatial intelligence algorithms like those shown in this video These algorithms are the key to scaling up applications such as autonomous Driving 3D gen and content Creation in summary fvdb is broader in scope than prior Frameworks can handle larger scale inputs runs faster and leverages core Nvidia Technologies fvb also comes with batteries included containing many examples of real world spatial intelligence algorithms