Texture Painting 3D Objects Using AI at Real-Time Live! SIGGRAPH Asia 2023

Key Takeaways

at sigraph 2023 we showed interactive PBR material authoring using AI from a simple text prompt our optimized AI system generates seamlessly tiling materials complete with normals roughness and ambient occlusions to give more control we also enabled creators to perform guided local edits using AI for example let's add some rocks to this area in the material unfortunately tiling materials get repetitive pretty quickly which at local details directly in 3D with our AI texture painter you can let me show you how in our Nvidia Omniverse extension first I'll select the ground mesh and then make sure it is paintable then I'll open up my sample image from this sample Rock texture image I can create a texture brush powered by AI let me use this brush to paint a stone path directly on the grass nice I can see you're making a stone path uh can you zoom in on it oh interesting the texture is not actually repeating uh how is that happening AI is generating every patch of the brush stroke as I am painting inference is accelerated by tension cores in Nvidia gpus so we can paint interactively that's amazing what if I have a particular kind of pebbles in mind but no sample image well we can generate an image using text to image AI let me try with this prompt how does this look oh those look good those are really good rocks sure let me start painting along the path oh nice this is uh looking a lot more interesting now and I can tell there's a seamless transition between the grass and the rocks and the uh path how does that work unlike traditional texture synthesis our brush uses a foundational AI model which has seen combinations of many different materials it can fill in a realistic boundary between textures often found together all right this is looking great how about we uh paint on a more interesting 3D surface how about this pagota over here yeah the pakota is looking way too new let's add some Moss to it sure let me pull up an image all right I can start painting now wow even with those curved surfaces and the various seams the brush is painting with really minimal artifact well this is because in our method area operates in local render space which is tangent to the mesh surface and not texture image space using nvidia's kin library for 3D deep learning we project the generated output to texture image Space by rasterization and texture mapping awesome so let's try and make something a bit more unique I was looking at that house and thinking it looks like a gingerbread house so how about we cover the roof with various delicious desserts sure that sounds fun we can use the tool to texture the roof completely from scratch let me start with some hazelnut chocolate oh I love chocolate that sounds like a good plan uh you know that brush is really small it's going to take us forever to cover the whole roof at this point H that's a good point but I can increase the brush size so that we can cover more aerial all right much better adjusting the brush size is really handy um but you know we don't want to go all Heavy chocolate so how about let's lighten things up a little bit with some frosted cookies sure let me pull up an image let me swirl this uh snow cookie around the chocolate like so oh wow I noticed the uh brush is actually staying oriented to the direction that you're painting that's really handy um let's let's give a little color to the roof now and maybe sprinkle in some rainbow candies that sounds good I don't have an image available but I can generate one with AI oh wow that's uh honestly looking a little bit too delicious the transitions there are great it really looks like I'm making a real gingerbread house yes I'm getting hungry too excellent well here we've demonstrated the first AI powered system that enables realtime texture painting on the surface of 3D objects this is just a first step that shows the PO potential of AI when integrated with fully interactive tools driven by artists and their creativity