Diffusion Texture Painting | NVIDIA Research

Key Takeaways

we present a method to apply generative diffusion models to interactive texture painting on 3D objects unlike existing texture painting systems where the same image is repeatedly stamped over the surface our method allows painting of any complex texture with variations and seamless tiling any inspiration photo even of lower resolution can immediately become an interactive texture brush with no training or pre-processing we can even generate a c texture using text to image diffusion during an interactive stroke we cast AR Ray from the cursor through the mesh and use normal and previous position to orient a local camera pointing at the mesh the local patch rendered from this camera becomes a conditional input to an imp painting image diffusion model the output of this generative model is then back projected into the texture image using standard UV mapping this process repeats for every patch of the painted stroke because diffusion supports in painting this allows generation of local patches that tile seamlessly with the texture already painted applications of our method include rapid 3D concept design for example prototyping a fantasy garden and exploring gingerbread house looks notice that patches tile with each other naturally with no user defined Alpha Maps or manually painted transitions designers can develop bold fabric prints for example using these painting [Music] Inspirations it is also possible to complete this realistic photogrametry asset using Forest textures sampled from photos captured in the wild notice how seamlessly different bark textures transition to each other or we can Source textures from this physical toy to develop a similar look on a different 3D toy notice how different yarn patterns are melting [Music] together Beyond generating RGB textures we prototype a multi-channel material decoder that allows painting additional PBR material channels such as normals roughness and ambient inclusions which result in more realistic rendered [Music] appearance we piloted our system with two professional 3D artists who found our tool easy to use and created realistic blending effects usually timec consuming to achieve in the existing software [Music] our method shows that generative AI models can be integrated into highly interactive artist-driven workflows where the artist holds the brush we hope to inspire more work empowering artists with interactive and controllable Aid driven tools [Music]