Ray Tracing Essentials Part 7: Denoising for Ray Tracing

Key Takeaways

I'm Eric Haines for NVIDIA and this talk is about denoising for ray tracing I have a quote today from Daphne color a Sanford professor who works on AI for biomedical applications she says the world is noisy and messy you need to deal with the noise and uncertainty this is particularly good for ray tracing because ray tracing can be extremely noisy on the left you'll see an image that has five samples per pixel using path tracing and you can see it's quite noisy there's a lot of black pixels that where light just didn't get anywhere near where we needed it to after 50 samples that's starting to look better 500 better yet 5000 it's looking pretty darn good and if you looked at really closely you'd still see some noise in there and in fact films themselves have this problem where they'll shoot 3,000 Ray's per pixel but they'll still see some noise so what they'll use and what we're going to use today is a process called denoising to make those images better and you can see there's a diminishing return here it's pretty much the variance goes down with the square root of the number of samples so if you go from say four samples that's twice as good as just one sample nine samples that's three times as good as one sample but it's diminishing returns so we just can't afford to shoot 5,000 Ray's per pixel right now so we have to do something else and as they say that's de noising our reality is this is that we start with a noisy image if we're doing any kind of more elaborate effects and we have to try to get to a nicer image we'd love to get this kind of image but we often will have a crude noisy image and then we have to reconstruct so reconstruction is called denoising and there are various ways to do it here's another example where the left is noisy and the right is d noised this de noising process can be extremely fast it says identical time but it's almost identical time it's the blink of an eye and what denoising does is basically look at that area that the various surfaces and tries to use data both in color channel and any other kind of information they might want to use like the normal or the color of the texture that's underlying the surface and use that to come up with some kind of filtering process where it tries to fill-in tries to infer what's going on in the surface so you could denoise by effect for example so in this image we have a nice soft shadow given by that plant unto a floor but might be difficult to actually denoise this if you try to do it on the final image because the floor grain would possibly mess up your algorithm so what you could do is instead de noise just the shadow image which would just be a bunch of grayscale tones and then you would fold that in with the textured surface and get a nice final effect the problem with de noising by effect is that if you did this for every single pass it would start to really add up the de noise and costs would become exorbitant so what we try to do is D noise on the final image that way we have just one de noising pass so on the Left we have the original image and on the right we have a human filtered image and a neural network filtered image and the human one is basically using sort of traditional denoising techniques and then tweaking and so on and the neural network is using an entirely different process what's interesting here is that I think the computers are winning because in the upper-right you can kind of see that back wall there's a little strip of green it's kind of blurred out a little bit too much with the human version but the neural network is picked up on that and kept the vertical stripes they're deep learning can be used for an image de noising the way this works is that we have a bunch of rendered images 20,000 40,000 however many we can get as training data and then we train our neural net using those images to have the neural net kind of know what the environments like we can then use that neural net to take a noisy image and have it infer what the real image should look like so we have some huge training set or some reasonable training set it just depends and from that we then can actually do a great job of denoising images surprisingly good so here's a noisy image one sample per pixel and here's our D noised image so the shadows look really nice and notice how soft they are a little bit of soft shadow it's a fairly nice final image you can compare that to the ground truth they're almost identical in comparison the traditional method and rasterization is to use Shadow Man where you render everything from the lights point of view here you get somewhat sharper kinds of shadows they're just not as beautiful let's face it and it has other kinds of problems like that one person is floating a little bit it's called Peter panning and this can be avoided by using newer techniques here's another example of denoising where we have this shiny surface varying in roughness and denoise it looks pretty good and here's the ground truth now there's a fair bit of difference between the de noised and the ground truth here but it's enough to be plausible it's a reasonable result and it's one that is going to basically be reasonable to most people's visual systems they're not going to be surprised or shocked by the result in comparison here's one that's pretty different actually this is called a stochastic screen-space reflection method that uses rasterization it's kind of using information in this screen and it has problems there are ways that it works fairly well but there's other places where it kind of falls apart to show you the comparison again we have the ground truth and the screen space reflection and you can see they're considerably different here's another image here we have just one sample per pixel of raytrace global illumination and denoising we get this really pretty fantastic result it just blows me away that it can do this well to compare this to ground truth and the ground truth image you'll notice a little bit of darkening around the fringes of things and in the crevices and so on but for the most part the images are quite comparable last I'm going to finish off with a animation so there's a movie that was rendered called zero day by this person called people and he kindly put his entire database an animation path and so on on the web for people to reuse as they will so we use this at Nvidia to experiment with different denying operations this is a pretty complicated scene there's actually about more than 7,000 individual triangles that are moving around that are lights so those light sources are all moving around and moving light sources can be quite tricky to capture nicely and so in this video what you're seeing is on the Left you're seeing four samples per pixel and about 16 Ray's shot per pixel total they're bouncing around a bit and the D noised is on the right now this is not real time at this point it's about seven frames per seconds the calculation going on here but you can see that this noise result is quite nice here's the final result using denoising and if you want you can compare it to the original there will be links on the website it looks quite nice you have to really kind of freeze-frame and do a side by side to see where there are slight differences between this and the one where they traced thousands of rays per pixel denoising to me is magic to summarize it's just this cool technique that can work surprisingly well on cleaning up a lot of problems and a lot of under sampling that we'd love to have more samples but we can't and I think to me it's what really made ray-tracing jump ahead a little bit more quickly than people expected I think we also were thinking well ray-tracing eventually there will be hardware but denoising really takes a great leap you know instead of needing thousands of samples or hundreds of samples or even tens of samples we can get by with just a few samples in many many situations to conclude I'd like to have one more quote so we started this whole series with there's an old joke that goes ray tracing is the technology of the future and always will be well the futures here and I like this quote from Steve Parker which is ray tracing is simple enough to fit on a business card you're complicated enough to consume an entire career for further resources see the website ray tracing gems is a book I highly recommend given that I co edit it and it's free for download and I hope you take advantage of it and thanks for letting me have your time [Music] you