Adding an Interactive 3D Camera System // Ray Tracing series

Key Takeaways

hell gray is so much better than english breakfast what's up guys my name is the churno welcome back to the ray tracing series today we're going to be talking all about cameras and specifically implementing a camera system into our existing ray tracing code this over here being able to move with wasd keys and rotate the camera with the mouse this is what we'll be achieving here today now this kind of camera system that we'll be implementing here is going to be virtually identical to the kind of camera setup you would have for an actual real-time kind of rasterized graphics application so if we take a look at any game engine like unity or unreal or hazel the way that we'll be making our camera system work within our ray tracing project is going to be it's going to be the same kind of camera because we are technically sending out rays there are some shortcuts that we can take i mean we've been taking one already if you take a look at the code that we're running at the moment you know we're able to generate these kinds of coordinates and sure like they're not tied exactly to an aspect ratio but that's also pretty easy to do if we break down these components into like the math like what we're actually doing by shooting rays out into all of these different directions it's honestly pretty simple and probably doesn't require any any super advanced math unless you're trying to do something super precise which i mean we we will be remember the fundamental camera system is basically just a position so where is the camera in the world right is it here is it there because that's obviously going to define what the camera sees that is what we refer to as the ray origin but then there's also obviously where the camera is pointing is it pointing that way is it pointing that way just knowing where it is in the world isn't really enough because we need to know which way it's facing and that is what our array direction is now at the moment inside our code base we basically have a fixed ray origin which is zero zero one so we've translated our camera by one unit along the z axis and then we have array direction which is simply in the negative z direction so facing backwards compared to where we moved it to and then across some kind of defined field of view but really that field of view is going to be between like negative one and one because that's the coordinate that we're actually passing in over here using this method we don't have an easy way to rotate the camera and we also don't have an easy way to adjust the field of view and the field of view is basically what determines what kind of lens we actually have on our camera is it like a wide 18 millimeter lens is it a super like telephoto 200 millimeter lens we probably want to be able to control that and if we get into like super physically based aspects in this series in the future which we might then we would potentially want to like be able to customize the sensor size so how big is this kind of light sensor that's capturing all of this light because that's also going to determine the field of view effectively as is the actual lens that we can set so instead of setting like a vertical field of view of like 45 degrees or something we can maybe chuck on like a 50 millimeter lens on this ray tracing like application and then see like a realistic render from a 50 millimeter lens with an appropriate aperture with an appropriate depth of field relating to that aperture i mean you know the possibilities are endless but anyway the main kind of goal of the camera system that we'll be implementing here today is the ability to actually interact with the camera so i want to be able to like move the camera around use the mouse to like be able to rotate it i want to be able to interact with this application because obviously we can see that it's running in real time anyway it's like six or so milliseconds per frame on this hardware we should just be able to fly around it as if it was just a normal rasterized like renderer and that is exactly what we'll be doing here today now i already took the time to write the majority of this camera system this is the ray tracing series camera systems and input handling and all of that aren't really related to rendering so i didn't want to waste too much time in this series talking about that but we will go over what i've already written and we will have to integrate it into our existing code base as well as always though if you're not understanding all the math in this series and want to go a little bit more deeply then i highly recommend you check out the sponsor of this series brilliant brilliant is a website filled with amazing courses on various stem topics and their large collection of math topics is absolutely brilliant forgive the pun all of the math that we'll be using in this series from linear algebra all the way to calculus they cover all of it and unlike other course platforms their courses are extremely engaging interactive and visual instead of just making you watch videos and maybe giving you exercises brilliant will actively quiz you after like each section and present everything to you in a visual interactive form so that you can like play with the math and see the results of that it is absolutely the best way to learn all of this stuff learning it without any kind of visual aid can be extremely challenging aside from math they've also got a large collection of other courses on topics such as computer science data structures artificial intelligence the best thing about brilliant is that you can get started for free just go to brilliant.org the churno that link will be in the description below and you can get started right now but if you do like it brilliant have also been nice enough to offer the first 200 of my subscribers 20 off an annual membership using that link in the description below so definitely check that out huge thank you as always to brilliant for sponsoring this video so i have this camera.h and camera.cpp file these have been written using walnuts existing api but i've added some things to walnut which is the underlying framework for this ray tracing application so make sure that you pull the latest wallnaut code if you are writing this on your own and these two files are obviously going to be part of the commit for this episode on the raytracing github repository so that's where you can grab these files i'm going to copy them and coming into the solution explorer over here if i just go to the main folder of our ray tracing series let's go into source and then i'm just going to paste them right over here if i go back a couple of directories i should just be able to run the setup.bat script and now it will regenerate the project file which will mean that the camera.h and camera cpp files are here by the way as a little side note if you're not sure how to update a git sub module all you have to do is go into the directory of the sub module and because it's a sub module that kind of is a git repository in and of itself so if we just open a command prompt inside this directory all we have to do is type in git pull origin master to just get the latest master branch from the remote repository and that's it you can see that it's pulled the latest walnut code and then once that's done you'll have to rerun setup.bat to regenerate the walnut vcx project and all that with the latest code as well because there have been some files that have been added now if we go back into here and we reload everything let's take a look at the camera class so we have this header file and this cpp file i've tried to be relatively clean like with this api so that it makes sense this is moderately based off of hazel's camera system however i've simplified it quite a lot i've opted for kind of like an unreal engine style camera i guess unity has this as well i think where you can just hold right click and then move the mouse to rotate and then while you're holding right click you can also use wasd keys to fly around so it's kind of like a first person camera rather than like an orbiting camera and my major kind of reason for doing that is because on laptops without like a middle mouse button i find that kind of camera is really easy to use if you if all you have is like a trackpad so hopefully this will be good for most people now using this class is pretty simple it just takes in a vertical field of view this is in degrees probably important to specify that and then we also have a near clip and a far clip plane now these refer to basically the near and far clipping planes of what our camera can see because what happens is essentially like a frosting will be created out of this field of view and these near clip and far clip planes and anything outside of that frostum will be clipped it won't be rendered now without getting too deep into this the reason why it will get clipped is because eventually we're going to take all of this like data and convert it into that negative one-to-one space that we've been dealing with anyway and anything outside of that space just simply will not end up on the screen so it works that way both for rasterized graphics and also over here inside our ray tracing project we have to call on update every frame with the time step this is what's going to enable us to move around at like a constant speed that is independent of frame rate on resize is going to be important as well because we'll have to recalculate our projection matrix and then we have a bunch of getters for various details of this camera that we might want to access like the projection the inverse projection the view the inverse view the position of the camera as well as a directional vector of where it's pointing then we have something a little bit interesting which is get ray directions in a minute we're going to take a look at the implementation of this and there is some decently heavy math that needs to be done in order to convert like all of our cameras projection matrices and view matrix and all that stuff into actual ray directions and into that kind of negative one-to-one space that i talked about earlier now doing this on the cpu especially single threaded and potentially without ssd instructions i don't need to take a look at if glm's configured to do that at the moment but it can be fairly slow eventually when we move this stuff onto the gpu this is going to be so negligible that there's just not going to be any dent really in like the matrix math the matrix multiplication map for example that we'll need to do in a minute here however i found that on the cpu single threaded the math does take like a few milliseconds per frame just to calculate this stuff which is why what i've done here is i've just cached the ray directions they still do need to be recalculated every time the camera moves however if you're not moving then you'll just be able to use the cache directions and it will be much faster then we also have a rotation speed this might be useful if you want to like adjust your mouse sensitivity which just this doesn't exactly support but we could easily add it okay and then we have a bunch of variables obviously so getting into the meat of this let's take a look at how it works so initially we just set up the direction the position to these arbitrary values which is kind of similar to what we were using we had negative one i think and one this is just zoomed out a little bit more i think the reason why i ended up setting it to three is just because uh the field of view that we pass in depending on what you pass in that's going to obviously dictate how like zoomed in you are so therefore like how far back you'll have to move the camera to compensate for like using a longer lens if that makes sense that's going to be dependent obviously on the field of view that you pass in you can technically measure the field of view if you wanted to and be like okay that's how many that's how much i want to see like you know in meters this far away from the camera but we're not going to bother with that okay so on update is basically going to grab the delta of the mouse now this is effectively the sensitivity which i've just arbitrarily set to 0.002 i think that's what it's set to in hazel as well it's going to subtract the current mouse position from the last mouse position so the mouse position from the previous frame this is going to give us a delta of how much the mouse has actually moved in the course of one frame we need to make sure that we're constantly updating this so that our last mouse position is accurate but otherwise if the right mouse button is not pressed then we just return right because this camera is only active if we're actually holding that right mouse button so if it's not pressed we want to make sure we return the cursor mode to normal and return otherwise we're going to lock the cursor now what this does is basically locks your cursor to the window so that you basically can't really move the cursor you can still get the delta but when you unlock it you'll find the cursor in the same position that it was before and it also hides the cursor which as you'll see will be quite nice when we want to move around the camera and not like look at the locked cursor then we have a little moved flag so that we can tell whether or not we need to recalculate those ray directions and also the matrices we're calculating a right direction so what is like you know if our camera is facing this way what is the right directional vector from our camera we're doing that by just crossing our forward directional vector with our up vector here and actually reading this code i'm realizing that since i also have up direction i might as well just use that here but basically what that will do is grab the up direction of the vector which i wish i had a pen here so we have this directional vector that's facing up that's going to be 0 1 0 so 1 on the y axis and if we cross that with a forward vector we're going to get a vector facing that way and that's going to be our right direction and that of course is something that we'll be using here in a minute now up direction is self-explanatory that's just a vector pointing up now because we don't support tilting here it's always going to be that if we did support tilting so kind of rotating along the z-axis then we would have to also calculate the up direction but otherwise it's a lot easier obviously to just set it to 1 on the y-axis we have an arbitrary speed here which should be adjustable and then we have basically all of our movement based on the input so if we press the w key to go forward we simply are going to add to the position our forward direction now i i'm actually going to rewrite a bit of this code now that i'm looking at it because this was very very hasty with my writing of this and i don't really like the way that i play this out what i'm going to do is just quickly change it while explaining it to be more like this and the reason being is that the forward direction is the main thing we're adding right so i want that to be the first thing we're looking at and then speed is a multiplier that goes on to forward direction and then really everything has to be multiplied by the time step so that we scale it appropriately depending on how fast our program is running so in my opinion like this is a much cleaner way of writing it because you can clearly see okay you're taking position and you're adding forward direction and the rest is kind of like whatever they just modifiers on that vector let's go ahead and change that everywhere okay so obviously the w key moves us forward then the s key moves us backwards the a key moves us in the right direction that we calculated up here that's why we need it uh the d key well the d key moves us in the positive right direction and the a key moves us in the negative right direction so we go left essentially q and e will just move us in the up direction and in the down direction so q will go down here and e will go up i'm actually not sure q and e which one's supposed to go up well whatever but either way everything here sets moved to true so that we know to recalculate everything now we don't just need to recalculate everything if we move using like the wasd keys and the qe keys we also need to recalculate everything if the mouse rotates right so if we move the mouse while holding the right mouse button then we need to well rotate the view but also mark moved as true so that we can recalculate everything and that's what happens over here so we work out how much we're supposed to move in the pitch and the yaw so the pitch is like our x rotation so along the x-axis which is our horizontal axis how are we rotating so essentially rotating up and down that's called pitch that obviously is tied to the y kind of movement of the mouse and when we move the mouse up and down we would like to pitch up and down now the your is going to be our x mouse movement so moving from side to side means that we your which is rotating around the y axis hopefully this makes sense it's kind of like an aircraft i guess when you're moving an aircraft as someone who used to fly planes this makes sense to me but hopefully it also makes sense to you it's quite common to use like pitch and your to describe camera rotation now then we have some scary looking code here because we're using a quaternion but basically all this is doing is calculating a new forward direction for us by taking in the pitch and the your delta into account so this quaternion really just represents the delta of the entire rotation so in all axes how much did we rotate and then if we just rotate the forward direction by that amount we'll get a new forward direction which is what's happening here we then mark moved as true and if we have moved then we do need to recalculate the view and the ray direction so looking at recalculated view that just builds up a view matrix this line by the way is completely unnecessary so we're using lookout for that because that's probably the simplest cleanest way to do this we're basically saying that the camera is at m positions this is the same as that code that i removed earlier that will translate it to position but then the orientation of the camera is going to be position plus forward direction so kind of just giving it like a vector to look at doesn't really matter if this is like normalized or anything like that and it doesn't matter if it's scale it's just like hey you're over here look at this coordinate please and that's what the lookout function will do for us and then we also have to specify an up direction which is just going to be 0 1 0 and finally we're going to calculate the inverse view by just inverting that matrix and that's going to be useful to us in the future now recalculate projection is also a function that we don't need to do here but if we do resize the camera so we have a an on resize function which you can see will recalculate everything view technically doesn't have to be recalculated i think it's really just projection and ray directions because that's gonna be based on the projection and the view but since view is just based on the position and the forward direction which obviously doesn't change we can get rid of this recalculate projection will create a perspective matrix for us based on a field of view which uses this vertical form over here the viewport width and height to calculate the aspect ratio and then also the near clip in the far clip plane and then we'll just invert it to get our inverse projection as well those are like all of our matrices calculated so now let's talk about the ray directions which is really the last part of this camera system as i mentioned this ray directions is basically a vector of vectors it's just going to be our kind of cached calculations so we have to do a bunch of math to figure out ray directions from this like view and projection matrix and you can see the way we do that is we calculate that negative one to one here as we did before so this code at the moment is pretty much identical as you can see to what we have in render a render but then we're going to calculate a target by taking the inverse projection and multiplying it to this so interestingly what we're doing here is we're actually going from negative one to one space back into like world space that's really what's happening here for those of you familiar with 3d graphics like using opengl or directx or anything like that you know that you have like a projection matrix a view matrix a model matrix which is like your transform and then like a vertex position and usually in your vertex shader what you will do is you'll take that vertex position you'll multiply it by the transform so that now you have like the vertex position in a world space because you've multiplied it by the transform then you multiply that with the view matrix and then the projection matrix and obviously if you're like in hlsl or in like row major you'll do the opposite so left to right your multiplication will be like vertex position times transform times view times projection versus something like opengl or vulcan which uses column major math mostly your glsl code will look something like projection times view times transform times vertex position what that does is that takes us into this kind of like normalized device coordinates like clipping space which is negative one to one that's what that math does here we know the normalized device coordinates we know the negative one to one because those are our pixels so what we're trying to do is the opposite we need to cast our ray in world space but we have negative one to one so we basically need to reverse this operation we need to take everything and multiply it by the inverse of each matrix and also do a perspective division as you'll see here in a minute to get us into world space so what this is doing and target is just really like our ray direction it's just an intermediate vector here but it's like where we're targeting this vector so we have the inverse projection times this coordinate and then we're multiplying the inverse view matrix by this target with a perspective division that's been normalized and that is now going to get us into world space and then we can simply cache that ray direction so this kind of math over here is really the reason why we're caching all of this normalizing and this matrix multiplication tends to take a little bit of time on the cpu again normally matrix multiplication should be fairly quick if you're using simd because it should be able to do it in way fewer instructions however i don't know if glm is using that here i don't know what you guys are running so it's just safer to cache this but when we do move this exact code to the gpu and it runs inside a shader whether that be a compute shader or a ray tracing shader like a rayjan shader that will just be very very fast so there's absolutely no reason to catch this in the future it's just for now and i want to kind of stress that okay cool so we've finally gone over our camera class made some adjustments let's talk about how we can actually use this inside our code base now so back in walnut app.cpp which is really like the main kind of entry point of our application we have this example layer where we do everything everything stems from here what we're going to do is we're going to add a new function here called on update and this is going to be an override because it's going to be something that we override from layer specifically this on update function make sure you update your walnut sub module because this is a new addition so let's put in that time step here and the idea is we're just going to update our camera here speaking of which let's create a camera now i don't want to create the camera inside the renderer because the render i shouldn't really own the camera i don't want the renderer making decisions about where we render our scene from that should be something that we parameterize and pass in so in other words when i actually call renderer render over here i want to be able to pass in a camera which will act as a viewpoint to render from that way if i want to render from like multiple points of view or i want to move the camera using some kind of like animation or whatever that can all be external it does not have to interact with the renderer it just gives it a viewpoint to render from so over here then inside our layer is where i'm going to create that camera now let's go ahead and include camera and then if we scroll down a little bit more you'll see that we do need to provide a vertical for veneer clip and far clip and i'm going to do that over here inside the constructor so let's go ahead and write camera here i'll write 45 degrees as my vertical field of view that's a value i like using near clip we'll leave it 0.1 and far clip for now we'll set to 100 but we might need to extend that in the future those are fairly reasonable values okay cool so our camera is set up let's make sure that we update it by calling on update with the time step every frame over here and then going down into renderer render i want to pass in that m camera now as a side note i also need to resize the camera so i'll just duplicate this function and add in camera dot on resize and then we're ready to render with the camera so over here inside the header file inside our renderer i'm just going to pass in a camera it'll be const camera reference camera like that we'll be sure to include camera as well i'm going to copy this and go to the implementation paste it over here so that we can access it and now basically what we need to do as we loop through every pixel here is just retrieve the appropriate cached ray direction because that should already be there now what about the ray origin well as you can see that's just a fixed value so the ray direction is really what changes per pixel the ray origin you can see stays constant for the for the whole image because our camera is not moving in the middle of us rendering our image in fact if we do that we'll get some kind of tearing because half the image will be from one point of view and half the image will be from the other so that's obviously not what we want so basically what we can do is we can get the camera position outside of this for loop camera position and in fact i'm going to name this our array origin so that we kind of tie this back into ray tracing and then the ray direction which we can access with get ray directions this is going to return a vector for us right this returns a vector and we'll need to provide an index as to which ray direction we actually want to access and that's going to be our ray direction now this obviously is going to be per pixel and specifically it's just going to be the pixel coordinate so x plus y times our final image get width and that's going to be our ray direction now that means that we don't need this anymore because we've already done that calculation inside the ray directions but it also means we have a slight problem when it comes to our per pixel function because that wants to take in a coordinate that we've now gotten rid of but that's okay what we're going to do is refactor this a little bit we're actually going to change this function name to be called traceray instead i'm going to go back to the header file make sure that per pixel is changed into traceray and over here as well so we've just changed the name now but then i also wanted to take in a ray instead of taking in this vector coordinate now what is array array is something that has an origin and a direction so what i'm actually going to do is go into our solution explorer over here inside ray tracing make sure i'll show all files here under source i'm just going to add a new item it's going to be a header file called ray and this is just going to be a very simple data structure here that represents array so it's going to be a struct called ray which will have an origin and a direction and that's it now we are going to add some utilities to this file in the future but for now we're just going to write this and that's it going back into our renderer header file i'm going to include this ray header file and then that is going to be what we pass into here now we could pass it as const reference it's 24 bytes of memory so it's probably better to not copy it necessarily but again that you know your mileage may vary because it depends on the architecture and everything and what the compiler decides to do but we'll pass in a constant ray over here that's going to be our ray origin and ray direction so we obviously have our ray origin and we do have our ray direction so what i'm going to do now is essentially construct this instead of doing it here though what i'm going to do is refactor this code a little bit so that we actually make the ray up here and then we can set the origin to be the position up here so we'll get rid of this and then the direction we're going to set over here so ray direction is what this is going to become and then we're going to pass the ray in like that in the future we might actually change this to also kind of act as like our rate cast struck so in other words it will contain like potentially a minimum and maximum intersection distance and this is also going to be basically exactly how we do this when we move on to doing this like all rtx style and ray tracing shaders okay cool so there we go we have trace ray it takes an array and returns a color for us so let's go ahead and now change this code so instead of using ray origin array direction here we don't need these anymore because we have our array.direction so we can just add a little dot here which is nice and ray dot origin as well so going through our code adding all of these things here just fixing all of the errors and that looks pretty good so now let's go ahead and run this and see what happens alright so we see an image much like we do before that's good news and then if we hold down the right mouse button and just move the mouse you can see we can actually move the camera and if i use the wasd keys check this out i can completely orbit around it and move around q and e will go up and down i guess e should be going up because that would be weird if q went up right this feels more natural to me at least but yeah you can see our render time has slightly increased especially when we like move but otherwise it's still absolutely not bad and still pretty interactive depending on of course what resolution we make all of this as well so yeah that's our camera system so now we can actually fully move around this sphere and see that it is in fact 3d and this will be super useful when we want to inspect like obviously our scene and in fact i mean at the moment we're rendering this all in real time this will obviously dramatically slow down depending on what we're doing however we are going to also implement some performance optimizations run this on multiple threads and eventually move it to the gpu so really it won't probably won't matter too i hope you guys enjoyed this episode of the ray tracing series if you did please don't forget to hit the like button below as always all of this code will be on github under a commit labeled episode 6 or whatever episode this is next time i think we might take a look at how to render multiple spheres because at the moment we can just render one and we might take a little bit of time to customize the ui so that we can like add and remove spheres position them using imgui and do all that fun stuff thank you all for watching definitely check out brilliant in the description below if you want to learn math and support this series and i will see you next time goodbye you