NVIDIA’s New AI: Painting With Light!

Key Takeaways

Here you are looking at a real video. Now this is not real anymore, but with new completely different lighting dreamed up by an AI. What? How? Not with a game engine and not with complex 3D software. This was done by an AI in one go. And the secret behind it is absolutely ingenious. I cannot believe that this actually works. So this is the regular workflow to create something like this. But scientists at Nvidia in this collaboration say, "Forget all that. Not necessary." No sir. Instead, you just take a real input video slap a new environment on this sphere and bam, their new AI recreates the whole scene as if the cat were really there. This is stunning. But it gets even more stunning. I still can't believe this is actually working. You see, this is the input. And now it is relit with neurog gaffer a research paper from just a year ago. Not much good is happening here. This other work from also around a year ago. This does not give me believable results either. We cannot see through the plastic bag. Now are you kidding me? This is diffusion renderer. We just talked about this a month ago and you are saying that the new technique is already significantly better. That is impossible. But let's have a look anyway. Holy mother of papers. Are you seeing what I am seeing? It is so good. I can barely tell that this is not reality. This was cutting edge just a couple months ago. And now we got this. What in the world? Seriously, my shoes don't go out of fashion that fast. So, how do they do all this? How does this black magic work? Well, when the AI technique looks at your video, first it tries to understand what part of the appearance is lighting and what is material. It tries to separate the two into something we call an albido map. And look at how accurately it captures even the super thin structures like hair. This truly is a sight to behold. And now when we add a new environment like this one, look, it can apply the new lighting to the material. Sounds simple, but this is super tough to get right. Let's see how hopeless this really is on a tougher scene. Oh yes, this one will do. Tons of shiny specular objects and to that I say not a chance that this will work. Previous technique, nope, nothing is happening. And then something is happening, but there are too many artifacts. And finally, the recent previous technique gives us choppy results. And oh my, that is what I was afraid of. This is believable. If you believe that glass can suddenly turn into plastic, this is Tupperware party. Come on, man. I want my whiskey bottles back. So, new technique. Oh, look at that. I can't believe this. Almost perfect. Loving this. This will be fantastic, for instance, to train these shiny new self-driving cars because you can create a ton of different variations for the same scene to make the car as resilient against these changes as possible. Make sure it doesn't have a meltdown the first time it sees a sunset or a particularly sparkly disco ball. Make sure it has seen everything. Or you can also put yourself in a video game world. And it works on a variety of realworld scenes. You see, previous techniques are not quite up to the task. And this new one seems nearly unbreakable. And this kind of improvement in just a few months. That is absolutely incredible. Bravo. What a time to be alive. But wait a minute. The paper says that it looks at 150,000 videos, but they don't have material information in there. Learning from that is not just difficult. That's mission impossible. And yet they pulled it off. Sorcery. So how did they do that? Dear fellow scholars, this is two minute papers with Dr. Kohaa. Well, they invented a clever workaround to get around this. Now, hold on to your papers, fellow scholars, because they run a pre-trained inverse rendering technique that guesses the materials for each image. It is a bit like karaoke. Yes, imagine that you have a great song that you want to sync to, but it contains both the instruments and the vocals. You don't want that. What you want is a song with the vocals stripped away to only hear the instruments. Just like this method strips away lighting effects to reveal pure material properties. Loving this. They call this auto labeling. But is this really important? You bet your papers it is. Look, if you skip that, o, you see that this version does not have that proper understanding of the scene and materials and gives us these ploty results. And if you add autolelabeling back, now we're in business. So cool. Here you see me running the full Deepseek AI model through Lambda GPU Cloud. 671 billion parameters running super fast and super reliably. This is insane. I love it and I use it on a regular basis. Lambda provides you with powerful Nvidia GPUs to run your own chatbots and experiments. Seriously, try it out now at lambda.ai/papers AI/papers or click the link in the description.