Problem Solving for Beginners: Build Algorithms from Scratch with First Principles and JavaScript

Key Takeaways

Learn how to problem solve with JavaScript. This course from Dr. Radu will teach you how to build software solutions from the ground up using first principles. You will learn how to identify sub problems, how to prioritize them, and what to do when they are interconnected. All algorithms will be implemented from scratch and explain step by step using simple language and visuals. So, this course is perfect for beginners to sharpen their skills. The focus is the problem of vision-based positioning systems because it has many layers. And this course is offered by Caria University. Check the description for how you can register for university credit if you get in before the end of October 2025. Hi, I'm Radu and you may remember me from such things as the self-driving car no libraries JavaScript course or the no blackbox machine learning course in JavaScript. And I just finished painting my balls so I can teach you something new. You see, I put them to the wall like that and used this pen with a camera on top to draw some secret things. Question is, can we reconstruct those drawings just from the videos? Spoiler alert, yes, we can. Now during the summer I organized this as a competition and everyone did amazing work. You can find all of it on my discord the discussion source code anything you want. And in this course we'll study this problem but my goal is to teach you how to solve difficult problems like this in general. I chose this one to focus on because it has many layers. It can be broken down into many smaller problems using the right techniques. I'll teach you how to identify these sub problems, how to prioritize them, and what to do when they're interconnected, how to isolate them so they can be worked on independently, and how you know if you did it correctly or not. These things took me literal decades to learn and I'm grateful for all the guidance I had along the way. But I wish there was a more structured way to learn all that. That's why I'm making this course. Now, there are experts out there who can solve this problem by adapting an existing method. They master the terminology, the techniques, and practice them every day. In this course, we'll take a different path, building our own solution from the ground up using first principles. The goal isn't finding a quick fix. It's developing a thought process. Learn to use simple reasoning and the math you know from school to tackle tough problems. We'll explore different approaches. First by hand, then by automating the steps with simple but powerful algorithms. These will be implemented from scratch and explained step by step using simple language and visuals. So this course is perfect for beginners who want to start thinking like those experts from earlier. And the things we study here are used in the real world to solve many important problems. I give examples during the course so you realize you do become more capable while playing with this silly thing. I use JavaScript because it's easy to get started. You probably have the tools already installed. The syntax is easy for beginners and it's good at visualizing things in the browser. And if JavaScript really isn't your cup of tea, just use your favorite language to follow along. One more thing before we start. Just listening to me is not enough. You may think you're learning because what I'm saying makes sense, but things tend to come in one ear and out the other. So, I really hope you follow along. Pause the video to think when I ask you to and do the homework assignments. Do me a favor and take your time with this one. Maybe even experiment in the real world. Paint your own balls. There's so much you can learn from doing that. I typically end my intros with some kind of joke, but I couldn't think of any this time. Even as Chad GPT, and no luck. So, let's just begin. Before we start coding anything, it's important we understand the problem and think how to approach it. On my website, you can find a link to the videos I recorded. The drawings were secret in the beginning, but to make the problem more approachable, I told everyone the first is a circle. The website also explains the setup with the virtual demo you can play around with. The balls are all the same size but have different colors. These two are horizontal and together they all form an equilateral triangle centered here at the same height as the pen when held up straight. Every drawing starts from this location with the camera pointing at the center of that triangle. After that crazy things can happen. Now let's play the first video and we can kind of tell a circle is being drawn, right? We see how it starts moving to the right side and then further back until about halfway. It's on the other side now. Getting closer and closer back to where it started. But how do we figure this out? Try to analyze and write down your thoughts as descriptive as possible. Write as if you're explaining it to someone over the phone. Pause the video to do this exercise and I'll tell you my answer in five, four, 3, two, one. I can see the green ball on the right looks bigger than the others. So, the camera must be on the right side at first. About halfway though, the blue ball starts to look bigger. So, it's on the left side. Now there seems to be a kind of a balance to this right left motion. I I also notice all the balls appear to shrink in the first half of the video. So the camera must be moving away from them. Then in the second half they appear to grow back to their original size. So the movement seems to loop back to the start. I cannot say if it's a circle for sure, but it's definitely close. Mhm. Were you able to write something like that? If not, try again now after hearing Leonard's version. It's useful to have the skill to analyze and explain your thoughts like that. No matter if you talk to yourself, your colleagues, or some AI model. Now, by highlighting keywords, we see those apparent size differences are important. Things that are further away do appear smaller because of perspective. You can see it here. These are all the same size, but this one looks about half as big because it's twice as far. And this works in general. Like if we know how far away something is in the beginning and it moves to look a fraction of that size, we can estimate its current distance from us using this formula. We have a good intuition for this and that was clear during the competition. In the first few days, everyone used the apparent size difference between the blue and green balls to estimate this offset. But to get this other one, they used different things. Min and Toby measured the gap between the balls. Yu measured the size of their bounding box. Danny from Denmark used the size of the balls as such and did some more complicated math to figure out the offset. What do you think of these? Would you do something similar or do you have a different idea? Pause the video and think what might be good or bad about these approaches. When the timer runs out, I'll tell you my answer. I think what Minion Toby and Yu did are the simplest to implement, but they hide the bigger issue. Things are indeed as expected when moving directly in front of the balls, but they get weird when moving to the side. Notice how the bounding box is no longer square or how this gap is partially hidden by the ball in front. I'm not saying their approaches are bad. I think they can work. But unless we account for those things, they're incomplete and accounting for them is not straightforward. Danny's approach has the benefit that balls look the same from all angles. And what he does here is more involved, but it's not rocket science. Let me show you using this setup where instead of balls, I have these telescopic rods. They do the math for us. When contracted, they all connect here at this starting point. The balls appear the same size in the beginning of the video. But if we move to some random point within the video and measure the blue ball again, this ratio tells us we need to make the blue rod 25% longer. And this is how far the camera must be from the blue ball. Now, but this ball will look the same from anywhere on the surface of this imaginary sphere. That's why Danny used another rod. If we do the same thing for the green one, we must be where these rods connect. They're restricted now to this circle. And if we know the length of the pen and assume it's perfectly straight and touching the table, there's only one place it can be. Danny ignored the red ball like that and that was really clever in the beginning because he could focus on a simpler version of the problem and implement a quick prototype. He was the first one to get something working and that's important because he could quickly tell if his idea is on the right track or not. If we start using that red ball now, it just makes that rod fit here like this. It doesn't really affect much in the first video because for the circle I didn't need to lift the pen. In other drawings, it will play a bigger role telling us if the pen is touching the table or not. Anyway, repeating this again and again, we will get a drawing. To demo this quickly, I made a special ruler that measures this size as a percentage relative to the first frame. I also marked the rods like this, showing percentages here, too, so I can match what the ruler shows. In this way, I don't have to calculate anything. You could call this an analog computer in some sense. Derek from Veritasium has a video on the topic. He thinks they might be making a comeback. If you want to play with something like this, you can make a quick setup using a simple tripod like I showed in the first status report of the competition. Now, the math we're doing here is called trilateration. It means figuring out a position by using estimated distances from known points. It's how cell phone towers and satellites can be used to find where you are. The main difference is how the distance to those points is estimated. Here we look at those apparent size differences. For cell towers, the signal strength is used. It's known to follow something called the inverse square law. So, we can estimate the distance using it. Phones only show you the strength to the nearest tower because you don't really care about the others, but it knows those as well and uses the information internally. For satellites, there's a different approach. The signals they send contain information about where they are and when the signal was sent. Those travel at the speed of light. So we can infer the distance using this relationship. I'm simplifying here ignoring relativistic effects and few other important things. But all they do is make the distance estimate really precise. The rest of the math doesn't change. Now in practice more satellites can be used to get a better estimate because of small errors those imaginary rods wouldn't really connect in that case but if we allow small adjustments they will and you'll have a better estimate one from many different viewpoints. Anyway, I sidetracked a bit, but it's important you understand that these are fundamentally the same problem because by studying this, it means you also start to understand those other things which are arguably more important than playing with my balls. That's literally rocket science. It also means what we have here is a working positioning system. It's important because unlike others out there, I don't have the resources to do experiments in space and I suspect most of you don't have them either. But with this thing, we can study that same math here on Earth essentially for free. Speaking of math, Danny didn't code actual rods in JavaScript to solve the problem. He could have. A fun thing you could try is using a rigid body simulator and model the rods the same way I did in the real world. But nobody does that because we can solve this analytically as well. We are much slower that way. But computers can do it in an instant. You'll see when we get to that. But now let's code a quick prototype to see if this idea can be automated in practice. Coding with Radu. Let's code now. Have your editor open in an empty folder and let's create our first file index.html. This is going to contain basic HTML structure. So starting with the dock type and the opening and closing HTML tags and inside let's add the head section with a title decoding the drawings. Let's test the file by opening it in a browser. I'm using Google Chrome and it's empty for now, but you can see the title appearing here in the browser tab. To add things to the page itself, we're going to have to use the body section and we're going to need a video element. I'm going to give it an ID input video and a way to select which video to play in it. I'm going to use an input. Let's call it file input. And this is going to be of type file. And I'm going to make it so it accepts only video elements. Now, if you save this file and refresh the page in the browser, you're going to see this. The video element is actually here, but you can't see it that well unless you open the developer tools, which by the way should always be open when coding. From here, you can select the elements tab, and there you can see the input video and the file input. Nothing happens when you choose a file just yet because we have to connect the two. We do that using JavaScript. Let's define a script tag and inside I'm going to take the file input and check if it changes. We'll add a callback function to that. And I'm just going to pass the event here and type debugger. Save and refresh. And now when you choose a file with the developer tools open, it pauses where we type debugger. And we could inspect this event attribute here. You can see it has a target attribute. And if we scroll down, we're going to find files. It's a list or an array where the first element is a file. It has some information here like how big it is, when it was modified, and so on. So in our code, we're going to have to do that. We're going to have to reference the file from where we found it by debugging. And now I'm going to pass it to the input video source in this way. And then the video can start playing. Now save and refresh. Choose a file. And you can see what this create object URL does is assigns the source to this cryptic looking address. I'm not going to go into the technical details here, but basically it uniquely identifies the video on our computer. And then it starts playing. The reason it's not playing is because the debugger is paused at the moment. So we can press F8 or here or here to make it go. And there it is, the video playing. Now, we're going to start processing this video, looking at the individual pixels. And for that, we can't use the video element as such. We're going to need to work with the canvas. It has functions that let us do that. I'm going to define this input canvas here. And we're going to have to set it the same size as the video itself. We can do that only after the video has loaded. So we're going to listen for that event and pass a call back function here like so. We can now set the input canvas width to the width of the input video and do the same thing for the height. And after that we are going to begin a loop where we're going to draw the video again and again over the canvas as frequently as possible. That will happen in its own function. I'm defining it here. And to draw on the canvas, we first need the reference to the context, the 2D drawing context that we can use to draw an image coming from the input video at 0 0. And then to loop we are going to call request animation frame. And we pass this loop as a parameter here. And it's going to call itself again and again many times per second. It tries to do it 60 times per second. But if this processing takes some time then it could lag a bit. Now we don't need this debugger up here anymore. Let's remove it. Save and refresh. Choose a video. And you can see now the canvas playing same as the video pretty much. And the canvas does have the same size as the video. You can check these properties here in the elements tab. Now we don't need the same thing showing twice. So we can actually make the video invisible using CSS. I'm going to go up here and type inline CSS. Basically giving a style and setting the display to none. Save. Refresh. Choose a video. We see only the canvas there. The video is in the memory, but it's not appearing there on the page. Now, this canvas here is quite big. It doesn't fit on my page nicely. So, we can also make it smaller with CSS. And when adding more and more styles to the page, it's not a good idea to do it directly in HTML like this. We do it in its own file like style CSS and link it here like so. To create this file in Visual Studio Code, you can control-click it. And now we can write canvas and to the canvas elements, we are going to set the width to 400 pixels. Now save, refresh, choose a video, and we can see this canvas smaller there. But it's important to note that the original size is still that of the video. We are just making it seem smaller so it fits nice on the page. Now separating the code into files like this is what we should do for the JavaScript as well. Otherwise it becomes unmanageable. So let's type here a script referencing main.js. Control-click this to create the file. And I'm going to take the code from here inside here. Save. And it works same as before. Now we're going to start doing image processing to segment the image and isolate my balls there. We're going to do that on a separate canvas. And I'm going to be specific here and say that this is the input section. And down here, we're going to do the segmentation on a canvas with the ID segmentation canvas. So, we're going to get the input pixels from here and display some kind of output on this other one. Now, if we save and refresh, we're going to see this section here. And the canvas is there. It's just transparent. Let's go here and add a style to make these more clear even if they're empty. A background color and a black border. And we can see it here now. But it's not the same exact size as this one. So we can go to main.js here. Copy these and edit to say segmentation canvas width and height equal to that. Save. Refresh. And now these dimensions match. So we can start working with the image data. After the image is on the input canvas, we can get the data asking the context for it starting at 0 0 and all the way to the canvas width and height. And let's log this image data into the console. Save. Refresh. And in the console here, you're going to see a lot of these prints happening. Let's focus on this one. And see, it has the information about the image like the width and height. And then this data attribute. It's a long array of numbers. This is the same as the width times the height* 4. because these numbers come in groups of four. Now, this is nested here, but it's really just one long array of numbers. And each four correspond to one pixel, the red, green, and blue component. And this is the transparency. But all pixels in the video are opaque. So, they will all have this maximum value of 255. And to understand pixels better, we're going to use a microscope. You see, this is what we get here in the gray area. They are actually a little bit bright all components. But if I move this over the red ball, you can see a huge difference. And then the green ball and the blue ball. You get the point. Now before we do the segmentation, I don't like that we have to type input canvas like this everywhere. So one trick you can do in JavaScript is you can dstructure here to get just the width and height attributes of the input canvas and then here use this width and height everywhere. Now we don't need this log. Instead, we are going to segment the image. And this is a function that we're going to have to implement. Maybe here at the bottom somewhere. Let's say segment image with the image data. First thing we do, I'm going to dstructure again to get the width and height and the data from the image data so that I don't have to type image data dot everywhere. And then I'm going to take the context of this segmentation canvas so that we can display some processing on it eventually. Now the output data I'm going to define it as a new image data with this width and height. And we're going to loop the original data coming from the input canvas like so. I'm going to skip here 4x4. And now I can access the red component like this. And the green component like this, the blue component like that. And the alpha component like that. And if I want, I could set this output data data to these values starting at the I index. And then let's put this output data starting at 0 0 in this segmentation canvas just to test. Now save and refresh. And it's basically cloning the canvas at the top pixel by pixel. But we can do more things now. Like for example, you could replace here this G and B with R and R. And what we get now is the red channel. So white here means there is a lot of red and black should mean the rest. But because this is not pure black on the background, it has a little bit of red in it. My other balls are less red though. Anyway, let's check the green. And we should expect the same kind of output, right? Save and refresh. And um it's not as similar at all, is it? I mean, the green is still quite bright, but not as bright as the red was. And these are actually brighter than the background. Like there's a lot of green in them, especially in the blue one. I'm always surprised by how counterintuitive this feels. Like looking at the image above, I would say that those are green and red and blue and in the same amounts. But the computer thinks is otherwise and we're going to need to account for that. Let's check the blue as well. Save and refresh. And it's more like the red for sure than what the green was there. So now the idea to segment the balls from the background would be to threshold somehow to keep just the pixels that appear more bright in these individual channels. We do that by adding a threshold here. I'm going to set it to a parameter with the default value. Let's just say 100 and see how it works. And here after we get those original values, I'm going to define new values for red. If the original value is greater than the threshold, I'm just going to set it to 255, the maximum, otherwise zero. And I'm going to do the same thing for green and blue. So G and B. And this is going to be now new R new G new B. We're not really using this alpha. So I can remove it from here and just type 255 here. Now save, refresh, and um it kind of works for the red ball and the green ball, but the blue the blue and some green combines because of what we saw previously. blue having some green in it. And that's going to be kind of hard to fix. We can play with this threshold more. Like let's say put it 150. And now the blue looks okay, but the green starts disappearing from here at the bottom. We can try going maybe 125. And it's better, but still somewhat disappearing from here. Can try 110. And you're going to see that it's either one thing or the other. We're going to have to be a little more clever here. Let's inspect that green again. So, the problem is there is quite much green here, but there's much more blue. I mean, look, if we take the microscope and check it out, blue is much more intense. So, how about we calculate a difference? Like maybe we go here and say a difference for green is green minus let's be general and say red plus blue divided by two. I'm going to average these out. And now I'm going to pass this difference here to visualize. You can see that we get a much better result. So we can do the same thing for the others. Let's say a difference for red and the difference for blue. And now here we can check this red difference, green difference and blue difference and pass these values as before. So the new red, new green and new blue. Save. refresh. And we still get a little bit of a problem with the green, but now we can just lower this threshold more, like maybe 80. And yeah, this looks pretty good to me. Next, we can measure how big these balls are by counting how many pixels we detect in each of these colors. After segmentation, we can prepare an array for those. let's say RGB counts and set each of them to zero. And now here we can say let's increase this value by new red divided by 255. So this is going to give us one or a zero essentially. And we do the same thing for index one and two or the green and blue values. And at the end here we can return these RGB counts. So now here when we call segment image this function returns something. We can get the value like this. And let's log it in the console to test. Save and refresh. And this is the number of red, green, and blue pixels on each frame. They change all the time. These values are like the signal strengths in that cell tower example. You can think of the signals as how much light is reflected by these balls and the camera is the sensor picking it up. These follow the inverse square law. So we could think of it in this way too and end up to pretty much the same thing. But let's stick to the original plan and calculate the width. Now circles are more commonly represented by their radius which is half the width. And in practice, it doesn't matter if you use the whole width or half the width or a quarter of the width when calculating that ratio. Scalers like that simplify. So let's use the radius because it's the easiest to get when we know the area or pixel count. The parent radi are going to be calculated from these RGB counts. And in JavaScript, we could use the map function for that. It takes a function as a parameter that says what to do with these areas. So in this case, we want to return the square root of the area divided by pi. We could also use a normal loop or do this calculation three times for each of those counts. But I want to show you the map because it's very common in JavaScript and it can shorten things quite a bit. Actually, this can be made even shorter because JavaScript also supports something called arrow functions. So, we can just say that this area parameter turns into the square root of whatever our function is. So, we can write this in one line like so. And if we debug now with this apparent radi, we can see that these numbers are much smaller now, like around 100 pixels. Let's remove this log. It won't be needed anymore. To visualize the setup, we'll need to know the positions of all these things in a coordinate system. But there isn't one. So, we need to decide where to put it. And that's the question. Where do we put it? During the competition, some chose this point to be the origin. I think it's because of the symmetry. But James used one of the balls instead. That can also make sense. There's even a real physical object there marking the place. And as he pointed out, it can make some of the math easier. I'll show you later. Now, I will choose this point because then the drawings are 2D. Essentially, the pen can only go up in this positive direction. And there's also a symmetry here like I mentioned earlier. Now, for the orientation, we also need to make a choice. My favorite is this one because the drawings would be on the XY plane, but this time I'm going to go with AOSUS because it matches the default in 3JS and that will make some things easier later. I'm also choosing it because he won the competition. So, you're welcome. >> [laughter] >> In principle, anyone would work and there are pros and cons to each of them. But putting some thoughts into it may mean you make the problem easier. So, it's usually worth it. Go to the top to define some global values. Like the ball radius, it's three. The distance between the balls is nine. The pen height is 18. The initial Z offset is 80. And we can use these to calculate the initial camera position. I'm going to make this as a 3D point with X is zero. Y is at the pen height and Z is the initial Z offset. We can also calculate coordinates for the balls. >> [sighs] >> I'm going to solve them in a separate function because this calculation is more involved. And let's just implement this function at the end here for now. We're going to refactor things later. But first, it's important to get our idea into the code. And we worry about refactoring later. Now, this initialize balls function really is going to work with that triangle as the basis. So the length of the triangle is the distance between those balls. And now we can already calculate the left and right offsets. Like the left is minus this triangle length divided by two and the right is plus that triangle length divided by two. And now we also need the height of that triangle. And we can get it using a formula you learn in school. It's just a consequence of the Pythagorean theorem. In this right triangle, we have the height squared plus half the length squared is equal to the full length squared. Rearranging, we find the height like so. This reminds me of school when my friend Dino used to say that if right triangles are around, there's nothing to worry about. That stuck with me and I'm telling you this because it can give you confidence in the right moments. Now there's another right triangle here and it's similar to this large one because these angles are the same. So these ratios are the same and from there we can figure out how far the bottom is from the center point. We can write this simpler in terms of the height length. So bottom is 1/3 of this length downwards and top must be what's left. So 2/3 of that length upwards. Most of the math we'll need in this course focuses on these right triangles. So it's really important you understand them well. If you had trouble following what I just showed you, I recommend you take a step back and check this video out. In it, I explain the Pythagorean theorem in an intuitive way and show you a way to prove it using similar triangles. Now I'm going to take out the radius here from that global variable and use everything we got to define three objects with X Y Z radius and color. The first one is going to be the red. The second one is going to be the green. Storing here these ball properties essentially. Let me arrange this a little bit nicer so that these align. I think it makes the code much more readable. And now to visualize all of these properties, we're going to go and create another canvas. It's going to be in an output section. Output canvas. And now we have to specify here how big we want it to be because this is not depending on the video. And back inside of the main, I'm going to implement a new function to draw on this output. So draw output. Let's get the reference to this output canvas. And now loop through all my balls. Take out from there these important properties that we are going to use and begin to draw a circle. And then the arc is more general and we need to specify that we want to start at the zero angle and go all the way to 360 degrees. But like pretty much all functions in JavaScript that have to do with angles, they use radians. So 2 pi. And now let's set the fill style to that color and fill. I'm also going to go here and draw the initial camera position for reference. So, I'm going to take out the X and Z from this global variable. And let's use the arc again. I'm just giving it the radius of two there and black color and fill. Let's call this function at the top after we have the balls initialized. Save. Refresh. And you can see the output here. kind of these small things are the balls and um the initial camera position is there. So these values are in centimeters the way I measured them and now three pixels is like nothing. So these values appear very small and I have seen during the competition some people try to scale these up like maybe multiply by 10 that would still give kind of a small values maybe multiply by 20. Problem is it starts to be confusing where do those numbers come from. So I like to keep these as such and then go where we are working with the output and scale it somehow. I also want to move these to be somehow in the center screen here because now we just see part of the setup. So for that we can use these helpful methods. The translate can go to the output canvas width divided by two and height divided by two. And to scale maybe we just use five for now. Like let's see how it looks like. Save. refresh and the top view already looks nice. Actually, it's a bit confusing because from the top, the red ball should be the first we see. Maybe we do this loop in reverse. So, starting at balls length minus one, greater or equal to zero, and I - save and refresh. It's better like that. But actually I think we said we work with the simpler version of the problem. So let's remove the red ball from the visualization for now. And an important thing every time we work with these functions restore the canvas so we don't forget it in this kind of transformed state. So I'm saving the state before doing this and restoring. Now it won't do any difference at the moment but when you call these functions many times or in a loop then problems start to happen. So I always recommend remember to do this and to make things clear we can also write here that this is supposed to be a top view. I'm going to do that before we start transforming things. So let's just say top view. And I'm going to give it a small margin maybe five. And the text is actually cut a bit because of how the baseline is by default. I'm going to tell it to use the top. It's a little better, but kind of small. So, I'm going to set the font to 20 pixels aial. Okay, I think this works. And one more thing that could be improved I think is um the contrast between the background and these are is not that great. Some people might have issues seeing them properly. Let's set the background to white by drawing a white rectangle over everything. So we need the width and height and um let's dstructure here so that the code becomes even better. So, I take them like that. We don't need this here anymore. And at the top here, we start off by setting the fill style to white and filling a rectangle the whole size of the canvas. And we remember to set the fill style to black again, otherwise this text is not going to show. And now it's better. I think your first homework is to figure out the initial distance to the balls using geometry in this triangular pyramid. Scribble down how you got the answer and implement that formula in the code. You'll know if it's correct if the result matches what I'm going to show you next because I'll solve the problem in a different way using the ukidian distance. It's a way to calculate distance between points. We know their coordinates. So we can calculate these lengths and use the Pythagorean theorem twice to get the length of this diagonal. If you want to learn more about it, including how it generalizes to higher dimensions, I have a video on the topic, but I won't insist on it because JavaScript has a way to calculate it already built in. We will store this initial distance to the balls inside of a global array. So we store three values, one for each ball. And we can use the map again. So for each ball, we're going to return here the distance to the initial camera position. Even though this distance is the same to all balls. So we could use just one variable for it. I'm using three because it makes the system more general. If at some point we have a different setup, then it's going to work without any other changes. And this distance function I'm just going to add here at the end. From two given points, JavaScript has a function for calculating the ukidian distance. So we are done. Save and refresh. We can debug here in the console. So if we start typing that and see that the initial distance is 18 something which is a little bit more than the 18 Z offset. So it looks okay. We can also visualize these by passing them to the output. I'm going to take here this pass it to the output because these are going to change eventually. So there should be a parameter here. And what we'll do is inside of this loop going through each ball, I'm going to copy this and do the same thing but with this distance to the ball. I'm taking the component associated with this same ball. And then here I want just to draw the outline of the circle. So, stroke style and stroke. Save. Refresh. And there we have it. These lines are a little thick. Maybe we can set the line width to 0.2. And this dot is actually quite thick also. So maybe we make it 0.5. Okay, much better. And now we're ready to estimate the distances when the camera moves from there. So we'll need to know how big these radi are in the beginning in the first frame of the video. And I'm going to store this information up here initial apparent radi. Let's just set it to null. And the first time we get some radi I'm going to set that value. So if that value is not set it's going to become those apparent radi like this. And now the distances to the balls just an empty array. We're going to do a simple for loop going through those apparent radi like that and calculating that ratio. So what it was in the beginning divided by what it is now. And the distance is going to be the initial distance that we calculated multiplied by this ratio. And now we can pass these to the draw output function call. But we have to take it from here. I'm going to remove it from here and just paste it here inside of our loop. So it updates on each frame with these new values. Save. And now if we scroll down here, we see these circles changing. This uh initial camera position is still showing there for reference. But you can kind of see this intersection point moving almost in a circular circular pattern. And now it's going back. Yeah, I can already tell we're on the right track. Creative coders might be thinking now that uh hey, those green and blue pixels are close to each other. Maybe we can look for that in the image. And yeah, it would work, but it's slow processing all those pixels and you don't really get the most accurate result since you're limited to whole coordinates. We can solve this analytically in a number of ways like the circles can be modeled using this circle equation. If you want to understand it well, I have a video on the topic or you can play around with Desmos. I made a special page for you to try out. This equation describes points on a circle and you can control the center point and the radius. Now, if we have two circles, they might intersect like this, touch, or do none of these things. If you solve this system of equations, you'll either find two solutions, one solution, or no solutions, depending on which scenario you're in. Your second homework is to tell me which one is it. find the x and y for these and then consider these as parameters to get a general formula for x and y that we can use in the code. Meanwhile, I'll show you a geometric way to solve the problem. The same thing James did during the competition. We're going to go here after we measured the distances to the balls. And I'm going to use the same naming as in James's diagram. So, A is going to be this distance between balls. B is going to be the estimated distance to the blue ball and C is the estimated distance to the green ball. Using the Pythagorean theorem in this right triangle, we can write Z ^ squ to be this. We can also use the Pythagorean theorem in this other triangle and write Z ^ squ to be this other thing. Now these have to be equal. We can simplify to get the final value for x equal to b ^2 - a^ 2 - c^ 2 / 2 a. Now x is known so we can use it to get the z value. Just type z is the square root of c^ 2 - x^2. And these are our camera position in this 2D simplified version of the problem. Let's pass it to the draw output function call here and update our draw output to have this camera position as well. So instead of this initial one, it's going to change. Save. Refresh. And we can see here there is a kind of an offset to where the circles actually intersect. Now this works for James as such because he chose his coordinate system centered on this ball. In our case it isn't. So we have to account for that. And this is what I meant earlier when I said that sometimes choosing one coordinate system over another leads to simpler math or other benefits. In this case, I have to do more math to compensate for that. So adding here an offset. Save. Refresh. And now you can see this dot is exactly where the circles intersect. Now, this is almost the same thing I did with those rods in the beginning with one small difference. This 2D simplification forces the rods to connect here at this level. It means the pen gets pushed back slightly because the distances were estimated in 3D. It's not a big difference. So, let's not bother with it now. But, we did notice it. So, we do make a note of it. Maybe at some point it becomes important. Let's collect all of these positions that we find and draw a path. So, I'm going to go here at the top and say camera positions is an empty array. And every time we get a new one from this calculation, I'm going to push it to that array. And the output now is going to get this instead of just one position. So maybe we just call this path since it's going to look like a path. And here I'm going to loop through that path. And I'm going to use a for each loop just to teach you that these exist as well. And sometimes they can make things easier to read. They are actually a little slower than the normal loops. So be careful with that if you're looking for efficiency. But um I don't care much about that in this video. Now here we can get the x and z from this point directly or we could write x and z directly here at the top and remove this line entirely. But x and z need to be wrapped in parentheses to look like a single thing. And you can see the path appearing there. Now we now have something that looks promising. So, what do we do next? Make a choice. Pause the video and make a choice. It's a really important decision. I'm going to tell you my answer, so get ready for that. I know it's tempting to start analyzing this and figure out how to improve, but in that way, this spaghetti code becomes even worse, and then you're going to be tired, and you'll leave it like that. Even if you're the only one working on this project, you will forget many things about it before you open it again. And if you leave it in this state, it's going to take forever to get back up to speed. Fact of the matter is now is the time to refactor when everything is still fresh in mind. Now, we won't overengineer this. Just a minimum I think is needed to keep things clear. But if you are interested in software engineering, check out YSU's code from the competition. He did many clever things there like extending classes and other things like that. So to make the code better, we go up at the top here. Notice that all of these are parameters. These globals can go in their own file. So inside of index html, we're going to define more files here. Maybe parameters js. Ctrl click creates this in VS Code. And I'm just going to copy from here this and also the function to initialize the balls from somewhere down here. Now normally functions should have parameters initialize balls relative to what? So some of these properties should go in it. But I'm not going to engineer this too much because it's kind of clear that things related to these global variables happen in this file. Now let's save main.js as well and see if things still work. And they don't distance is not defined. This is happening in parameters.js now because this distance function that we define in main doesn't exist yet. The order in which you include these is important and in modern JavaScript you can go around this by using modules and more software engineery stuff. But this problem is really hard as it is and I'm not going to introduce those other concepts. So we just have to pay attention to that. And we can create a new file here called mathjs. And move that only function there for now. Save. Refresh. And the problem is gone. Select the video and see that everything still works. Next, let's separate this image processing stuff. So segment image. This function can also go in its own file. I'm going to cut it from here. Create a new file here. Image processing. More similar functions can go in it, but for now it's just this one. Save. And let's do same for this output. So cut duplicate this output paste. Now every time we do this separation, I also check to see if there's something I don't like about this code. Like now I see all this kind of duplicated stuff happening because drawing a circle essentially in slightly different ways. So maybe we can take out the function a helper function for drawing a circle on a context with these properties and then we're going to just call this function there instead. So simplifying the code and making it more readable. This is going to be a generic circle drawing code. Setting the fill style and the stroke style. And now we have to check is there a fill style defined? If there is, we can call ctx.fill. In modern days and not just JavaScript but other programming languages too, we can do something like this. So it will evaluate fill style. If this evaluates to true, then it considers what happens next in this and. If this would evaluate to false, then it doesn't consider this part anymore. So, it's like it doesn't exist. Of course, you can write this using if conditionals. But I want to show you this because it's common and it can help you out if you see it somewhere. So, calling this draw circle now is going to remove quite a lot of code. We can say here draw circle ctxz radius color and here we can say pretty much the same but after xc we pass the distance to that i ball and then no fill. So, no. And the same color that we used here. And here we can just say draw circle 0.5 black. Now, because this line is the only one here and it's quite short, we can write the whole thing on one line. And we can even remove these curly braces. Like if we remove them and also remove the semicolon here then JavaScript doesn't complain and we can just put this up here. I think this is quite clear. So for each element in the path we take its XZ component and use it to draw a circle that is black and has a small radius. Now this is okay I think. Let's close some of these unnecessary files. And inside of main, it looks much better already. Let's save and quickly test to make sure the changes we did still work. And they do great. So, next thing is here. After we get these RGB counts, we started being sloppy and just added a bunch of code to this function making it really confusing. Like of course you know what it does now, but imagine you come here after a week and try to re-remember all the things. It's time to make it clear. So using these RGB counts we get the radi essentially estimate the distances to the balls. So it makes sense that that can be a step like the distances to the balls will come from this type of function. And then when we have these, this part is James' solution to estimating the camera position using the distances to the balls that we just estimated and the ones from the parameters. So another step is going to look something like this. And this is all we need in this loop really. followed by collecting the positions and drawing the output. So everything else has to go somewhere else. And these steps are going to become very clear what really happens in the loop. We'll create a new file for all this calculation. Let's put it here. Estimator. And in it I paste what we had in the main earlier. And here we can start defining those two functions. So estimate distances to balls from a given RGB counts. And let's return here those distances. And the next one estimate camera position from distances to the balls. And here return camera position and remember to close the curly brace. Actually we can return also this directly because defining that variable just to return it doesn't make much sense in a function where the name of the function is essentially the same as the variable we just had. And up here, we could look at the math a bit and notice that dividing by pi doesn't really make a difference. It's just extra work. So removing it is fine. And this initial apparent radi, it's not clear where it's coming from. So we could take it here from the top of the main. Makes more sense to have it here where it's being used. Save. and refresh and everything still works. But one thing I don't like is that we always have to keep doing this like scroll up and down and up and down. And um may seem like a small thing, but if we fit this one here on the right, we don't have to do that. And that can save time when debugging. So let's do that. In index html, I'm going to make a new structure for this. and group the input and the segmentation in a div so that it's going to eventually be on the left side and then this output goes in another div on the right side. For this to work, I have to tell the body that it should go horizontally and it's enough to tell it to use this play flex this time. That's much better. And uh since I'm here, I can also change this font to be sans serif because it will make it look more like font here for consistency and I like it more. There's still this warning here appearing on the right saying that we can get the system working faster if we if we do this. So let's do this in main here. We just have to specify that we will be reading the image data frequently from that canvas and the warning is gone. One thing that is a little bit not so nice is that uh I see kind of a pixelation here and uh one solution to the problem is to just make this output c