Stanford AA222/CS361 Engineering Design Optimization I Probabilistic Surrogate Optimization

Key Takeaways

all right we'll uh go ahead and get started so good morning everyone I'm Joshua Ott I work in the Stanford intelligence systems lab with Michael and my research focuses on autonomous exploration so that ranges from using teams of robots or or any autonomous system to explore some unknown environment so that that can range from uh Rover exploring a planetary surface and looking at where to place its its drill measurements to collect samples of the surface or it can be teams of robots so aerial and ground robots in a search and rescue application trying to coordinate and identify likely areas where survivors will be and then going and exploring those spaces so as you can imagine that autonomous exploration relies very heavily on optimization and specifically what we're going to be talking about today with surrogate models and probabilistic surrogate models so I'm gonna pick up where we left off on Thursday on surrogate models and then we'll dive in to probabilistic surrogate models so if you recall from Thursday we were talking about surrogate models and we left off on model selection so we had already fit our model to some data and now we're trying to evaluate the the quality of that model so the data that we've used to fit it we call that our training data and so that's what that's what we use to create our model based on whatever data we've seen so far once we once we have that model we want to evaluate how good is our model doing at data that we haven't yet seen so how well is it generalizing to data outside of what we've just trained on so to do that we want to look at the generalization error and this this is giving us some Metric to say this model's doing well this model is not doing so well okay so that's that's what we're sort of after here so one way to to quantify the generalization error is we can just write it like this is everyone able to see this board here okay so we can write the generalization error as the expected squared error so we'll we'll walk through this here in a second okay so we have the expectation over all of the X's sampled from this space of x's so this is all of the the possible data we could see and then we have this is the true function value so this is what we're interested in right we don't actually know that function value so we have our surrogate model here and this is what our surrogate is telling us so so this is a good definition to work off of the only issue is that we don't actually know this function value so so it's a good definition to go with but it doesn't really get us anywhere right now because we don't we don't know this to start with so another thing that we can we can think about is the training error right so this is how well we do on the data that we just trained on so very similarly we can write that like this where we have we've seen these M data points so far and then we're just looking at the error on those specific data points okay so now this avoids this problem because we know this we've gotten these data points so we know the function evaluation at those specific data points and we're comparing that with what we predicted so that's our training error and this is good it gives us gives us some Metric the only issue now though is that just because we have a good training error does not necessarily mean that it's a good model right we might do really well fitting our our training data but we might generalize terribly to other data that we haven't seen so so that's still an issue that we have and and that's just the the main point here is that models with low training error they can still perform poorly on on data that we haven't seen so far so the the training error is not necessarily a good indicator of our generalization error so one way to to get around this issue is through using the holdout method so this attempts to estimate our generalization error by looking at uh we partition our data our data set into a training and test set okay so we have some we have some entire data set that we've collected and now we're just saying we're going to take our eighty percent of our data and we'll call that our training data and we're going to save the other 20 and we're not going to use that for fitting and that's our test data so so this is nice because we we basically still get access to this we're just saving saving it for later okay so we take our our blue partition here our training data and we train on that we fit our our surrogate model and then we're going to test on the the Red Data here so an example of this that you can see the Blue Points here those were our our training samples so we fit some model on on just those uh data points and then we saved the two the two red points over there for our uh test data and so now we can see well hey our model does not do very good at predicting those those data points over there so this is sort of an indicator for us now that okay our generalization error in this case is not very good so that's that's the uh the holdout method we're just partitioning the data saving some of it to to evaluate on so cross validation extends this idea we're still we're still partitioning the data set it's just now instead of just doing this train and test split we're partitioning it into K different subsets so in this case we have five subsets here each one we we just randomly partition up so one through five here and then with those partitions we're going to walk through this and so at at each iteration that we're going through we're going to use one of those partitions as the holdout set okay so if we if we walk through an example here let's look at this so we've we've partitioned our data up into these five different partitions and now with the first iteration we're going to go through we're going to hold out just the purple partition so just this D1 partition here we're holding that out we're going to train our surrogate model fit it on this data only and then we're going to evaluate it just like we did in the holdout method on our purple data and that gives us one generalization error estimate we then repeat this so now we do it a second time this time we train on everything except for the blue partition and then we use that blue partition to get another generalization error estimate so then we can just repeat that for the rest of our partitions and as we do that we end up with now uh K different estimates of our generalization error and so with these we can then just take the mean and standard deviation of that generalization error that we got and that'll give us a new a new estimate so some mean and sigma for our our generalization error and you might be saying well why why not just do the holdout method why do we need cross-validation well the whole point of cross-validation is that it makes us less susceptible to how we partition our data so in the holdout method you can think that maybe in certain cases the way you partition that data gives you a sort of biased estimate of your generalization error you might partition it in a way that you get a really good generalization error but if you had partitioned it you know just a little bit differently that would no longer be the case so cross validation removes that that dependence on the partitioning by doing this this random partitioning into different subsets okay so any questions on uh cross validation or holdout yes I guess um if we go through and we do it this way technically at some point we have trained our model on all the data right yeah yeah that's a that's that's a good point um so you will at each iteration you're going to be eventually accessing all of the data but the the sort of caveat here is that at each iteration you're you're not having access to the the entire data set you're only getting access to that specific partition at that iteration so it's still hidden to you and then you get that estimate so you might have really bad generalization error on one specific partition and that'll affect your mean and standard deviation here we have five different models which one do we end up using yeah that's a good question so this this doesn't tell us specifically which model to select it just gives us sort of uh an idea of based on on the model type that we were using so maybe we're just doing like a linear fit it tells us that based on how we partition that data with the linear fit this is sort of what we can expect our overall generalization error estimate to be all right so today we're going to be talking about probabilistic surrogate models and uh this is one of my favorite areas in optimization and there's two two main reasons for that the the first is just that they're extremely versatile so they're very widely applicable whether you're looking at uh geoscience modeling so looking at minerals in the subsurface and sort of modeling how they they change spatially or if you're looking at Financial modeling they're applicable there as well so anytime I see that where where something is is used in two very different fields that's sort of like a flag in my head like hey this is this is pretty cool this is important so that's that's one of the reasons that I I really like probabilistic surrogate models it's just the versatility the other reason is that I think it's just a very powerful concept of being able to quantify the uncertainty in your prediction and I think that sometimes gets overlooked in in modeling applications because you just have your model and it's doing a reasonable job of predicting what you haven't seen yet so you're saying okay great that's that's awesome it's working well but you're not you're not taking that uncertainty quantification into account so I think that idea is really powerful so versatility an uncertainty quantifications those are the reasons why I really like probabilistic surrogate models and my hope for the lecture today is to really give an intuitive understanding first and then dive into the math in the details so if you if you ever like lose that intuition of why are we doing what we're doing just feel free ask a question and and we'll dive into that so to start off with looking at today we're going to be going probabilistic surrogate models first what they are how we use them and then we're going to be looking at how can we use them for our actual optimization uh purpose so I want to start off first with just sort of zooming out looking at the everything we've seen so far all of the different pieces of optimization and where where this fits in okay so so today we're focused here on probabilistic surrogate models and surrogate optimization and you can see it's sort of it's its own sort of branch in the the optimization topics that we've covered so far so that's not to say that everything on the top is not important it's definitely important and necessary to learn about especially if you're going to be using this in your work but the nice part is that probabilistic surrogate models are sort of their their own own Branch here so my hope is that if this is the first lecture you're attending all quarter and you're just drop it in right here that you would still be able to walk away from this lecture with uh with a solid understanding and probabilistic surrogate models so that's my hope okay so let's get started here with just a refresher of why we care about surrogate models in the first place so in in many different optimization problems often the actual evaluation of the function is pretty expensive so the function that you're actually interested in optimizing getting those uh sample values are are expensive so for example if you're doing some lengthy Hardware fabrication process that requires you to build uh the actual Hardware design before you test it and get that function evaluation that's going to be an expensive process right it costs a lot of time and it costs a lot of dollars to go and actually get this this Hardware prototype and then test it so a good example of that is if you're building an aircraft and you want to test it in a wind tunnel you can't just go and build thousands of different aircraft prototypes before you actually ship one right so you have to have some way to predict how your your function will evolve based on the limited samples that you've seen so far and lastly maybe another example here is you're you're about to go train some new deep neural network and it has you know billions of parameters and it's going to take weeks to months to train you want to make sure that your hyper parameters in there all are all tuned up nicely before you go and spend five hundred thousand dollars training that that model so that's where the surrogate model comes in it's a surrogate over the actual function that we're interested in and so because we're talking about probabilistic surrogate models naturally we would think that there's going to be some probability involved here so we're going to start just with a refresher of our friend the gaussian distribution and then and then we're going to work up from there to the gaussian process so I'm sure most of us have probably seen the gaussian gaussian distribution at one point this is just the the expression here for it where we have some mean and covariance Matrix and in two Dimensions we get something that looks like this where these are just the the Contour plots of it so just taking the slices here and just looking at the effect that the covariance Matrix has so as we as we change that covariance Matrix it sort of changes the orientation and shape of our of our gaussian distribution here and so if we want to look at you know different different distributions with the gaussian you know we can have two jointly distributed random variables A and B and we can represent them like this and so that that allows us like any distribution we can marginalize out one of the one of the variables so for a general uh probability distribution marginalization is just integrating out with respect to one of those those variables so we can just integrate out the effect of of B on our distribution and that gives us just just a okay and then lastly we have the conditional distribution so this is just saying that if we already know the value of B we can say what is what is the distribution now that we expect of a so it's a given B and we just represent that here through the gaussian update equations so it's just uh we just have this given value of B and then we're using that to condition on that value of B and that will give us a new gaussian distribution for our a okay so that's just a refresher because we're going to make use of these in building up the gaussian process okay so for gaussian processes we're going to start with just a visual example here to to build that that intuition so we're going to start and it might seem a little weird at first but what we're going to do here is we're just going to plop our our gaussian distribution in one dimension and we're going to take it all along the the y-axis here and we're just going to take five samples from that so we get our our five samples here and we're just plotting the dimension on the x-axis so here's just one dimension so they're all just along that that First Dimension there and then we're just plotting the value of the samples along the y-axis so that's all we've done pretty simple we can then take the same exact idea and we're just going to build it up to higher Dimensions so if we go to two Dimensions now we can have our our two-dimensional distribution here we take again five samples from that distribution and now we're just gonna we're gonna plot the dimensions along the x-axis here so we take the first X1 Dimension and we plot it there and then we're going to repeat and we're going to do the same thing for the second dimension so we take the the values of the from the second dimension now and we're going to plot them corresponding to the second dimension so all we've done is just taken those values and plotted X1 on the x-axis and then X2 as well and their sample value is on the y-axis and then we're showing the covariance Matrix here so you can see there's nothing you know special about this we're just showing that there's there's some correlation here as well all right so now we can just rinse and repeat this up to higher Dimensions so if we go to three dimensions you can see we get some samples here and now we're just again there's nothing special here about this particular covariance Matrix we're using all we're all we're trying to illustrate is that there's just some spatial correlation now so the value of X1 is more closely related to the value of X3 or sorry the value of X2 then X1 is to X3 so the closer you are in dimension the more uh relationship you have and so you can just see that through the the covariance terms uh in The Matrix okay so we can oh yeah question maybe like the zero plus row should be assumed what yes that's true uh well so yeah this 0.14 should be uh 0.61 this one here is that what yeah yeah that's right okay so we can continue that up so in four dimensions we can do it in five six we'll stop at eight for for this case so we we have uh five samples now from an eight dimensional gaussian where we've just plotted them uh along the axis here okay and so maybe these are starting to look a little like uh samples from a function to you and if they're not then we'll we'll clarify it by going up even higher Dimensions but uh that's exactly the the intuition that we're trying to get to here so if we if we take this up to you know hundreds of Dimensions so let's go now to like 200 Dimensions we'll start getting these smooth looking functions okay and we still have that spatial uh correlation here and so all we've done now we just have you know lots of lots of samples now and we start getting these smoother looking functions and so that's exactly what a gaussian process is it's just a distribution over functions and all we're showing here is five different samples from a gaussian process okay so it's just distribution over functions and all we're looking at here are the samples uh from that that function okay so if we want to make this connection a little a little clearer for a multivariate gaussian we have it defined by some mean and covariance Matrix for a gaussian process we're just defining it by some mean function and some uh kernel function so we can sort of write out the structure that we're going to be working with here so we're just going to write it like this and we're going to break this down in much more much more detail as we go but just so we have a starting point here we have our our y's these are the points that we're going to be observing so our function values and we're saying that those are coming from a gaussian process which we're defining by some mean function here at the points that we've seen so the m m different data points that we've seen and you can think about this this is very similar to the mean Vector that we have with a with a gaussian and then we have some we have our covariance Matrix as well that might be cut off for you guys over there okay so we're just we're just writing it out for all of our M different data points and we're just going to be making use of this in the rest of what we're going to be talking about here okay so all we have we just have our our mean function specified on on our data points and then we're looking at the covariance Matrix here is what we built but we use this covariance function or this kernel function here and so you might be asking what what what is this what is this covariance function well that's sort of the modeling choice that you have but a very common one and the one that we'll be using a lot today is the squared exponential kernel so how we write the squared exponential kernel this is squared here okay so we have our our squared exponential kernel it's just just the negative exponential here and then we have some length scale parameter so so what this looks like if we were to draw it out it looks something like this where we sort of get this Decay okay and so the length scale controls how fast we're decaying there all right so if we if we want to look at some some examples of this so this is just the same exact structure that we've written and this is sort of the definition that we'll be we'll be working with here today so if we look at some examples this is using the squared exponential kernel and we're we're just looking at we've already seen the center one where L equals one now we're just looking at how how does that length scale change the types of functions that we're getting so as you can see that as we decrease the length scale We Get Much More Jagged sharply varying functions and as we increase it we get more slowly smoother varying functions so that's that's sort of showing the effect that the uh the kernel function has on the type of function that we're using and so just like that length scale parameter you can see it has a pretty significant impact on the types of function uh function samples that we get the kernel function itself that we're choosing has has an even bigger impact so if you want to look at you know we're not just limited to the squared exponential kernel we can use any any number of these these are just examples there's even more out there that you can use so you can see that the the choice here makes a pretty pretty significant impact on the types of functions you're going to see and so that's where it becomes your your modeling choice of which one to use and so I'll highlight two here we've already seen the squared exponential kernel uh the second one is the matern kernel this one down here that you can see and uh they're they're quite different right the return is much more Jagged and has those sharp sharp spikes in it whereas the squared exponential is a lot smoother and so the the reason I'm highlighting this is just one example is in the geoscience modeling often when you're looking at you know minerals in the subsurface they're not just the smoothly nice varying functions there's a lot of noise there with these Jagged discon discontinuities in the subsurface and so a common thing to do is take a combination of kernels so you can represent the smooth nature of the the spatial changes in the subsurface but you can also represent that that sharper varying uh term as well so that's just sort of an example to highlight how you can you you can take combinations of these and it's it's really that that modeling choice and you can use sort of the previous things that we've talked about like model selection to determine how your your model is doing here okay yeah now what is the x-axis yeah the the x-axis here these are just uh basically like the X's that you've seen so these are the X's that you've seen not necessarily where exactly you have data but just where you're querying this this distribution at so we have some distribution over our our y values and so we're sampling the function values which is the the y-axis and we're sampling those function values at these these X points yeah previously when you had the uh I guess like uh in your board is M the number of variables or team number examples yeah n is the number of data points that we have so this well okay so this m is our mean function but this m is the the number of data points that we have seen so far so we have M data points that we've seen and we're going to get into a little bit of that here in a second so it should clarify that it's going to be working variables and all this I believe like it was increased increased size when you add in more variables right yes that's children if you like the number of variables the number of bits yeah it's the number of variables by the local variables yeah so so I think the example you're talking about was when we were sort of building up the intuition here yeah so this is this is slightly different and we'll see it I think in the next couple of slides um but just for constructing so this is on what we've seen so far these are the the samples that we've seen um so it's not an exact one-to-one map to that intuition that we were building up but I think you'll you'll see it uh coming up here and if you don't then then just ask again and we'll clarify yeah we are using like this kernel function for controlling each of the elements in the equivalent Matrix so which means that um this kernel function is approximating the covariance between two but how like it doesn't make sense to me that we have a covariance between two data points like Health like what does it mean to have a covariance between two things yeah yeah that's a great question so if we take the the spatial modeling aspect of it right um so say that we have and actually we'll go with uh like phone prices okay so say that you have we'll go here say that we have uh we've got prices here and we're looking at the number of sales that we have for those phones so maybe so far this is just based on like historical data here that we have we've we've sold the phone at these different prices and then we have data on the number of phones that we sold okay so the uh the spatial relationship there is just sort of telling us how much we expect the function to change based on how the the price relationship changes so I think this one is is intuitive right as we as we lower the price we expect to sell more phones because the price is is lower up to a certain point right so uh what we can do here is we can we're basically saying that uh the covariance between let's say this price here we'll call it I don't know P1 and this price here P2 we're saying that those are going to be more closely related than P1 and whatever this point P3 here so that's that's sort of the the intuition to have is that the the closer the data points are and that distance metric is another thing that you can change but let's say just in euclidean distance here the closer those are the more related their actual function value will be okay so here I think we're just going to build sort of a mental model picture of what we're going to be working with so the black line here is the true function value so that's what we're actually interested in that we can't directly observe well we can take we can take samples of it but in this case it's a noisy sample so that's why our black the black point here is a noisy sample of that function so all we have right now is one data point and we can based on the squared exponential kernel that's what we're using here we can then uh make predictions and then sample from it so those those uh blue lines that you're seeing those are the samples from the function from the gaussian process and so we can then condition on more samples so as we take more measurements we can make updates here so we can update our our gaussian process conditioned on these on these new measurements and so as we go with only three data points here again there's they're noisy so we're maybe we have some some sensor noise so we're not exactly measuring the true function value and that's why there's still uncertainty around those points but we can basically quantify the uncertainty there that we have around our our true function and so obviously this is you know with three data points we're doing quite a good job here a lot of real world functions are not this smooth and don't look like that but this is just to sort of have that picture of how we're we're progressing and adding more data points to our gaussian process and it's not just limited to one Dimensions so you can take this up and and do it in higher Dimensions as well so yes previous slide I I was curious what all the different blue lines are representing when you were showing the different types of kernels yes it seemed like the different lines were the models you would get for each design variable maybe that's wrong but I don't know if we only have one or two design variables here then what are all the blue lines yeah great question so the blue lines are the uh samples from our our gaussian process so just like in a uh you know like a one-dimensional gaussian you have something you know you have your your bell curve like that and you can take you know five different samples from it and get something like that that's that's essentially all that we're doing those are the blue lines here so we're just sampling you know I think like 10 here or something and the gaussian processible yep yep yep so yeah we're we're going along here we have like 200 query points to make this plot um and at those at those query points is where we're we're asking for the function value and we're gonna I think that's the very next one here so we can do this in two Dimensions as well and now I think this this will hopefully make things a little clearer so we've seen just like the build up of this structure but now we're gonna actually get to the prediction here of how do we how do we use these to make predictions about about the function okay so this example that we just saw uh we're gonna sort of introduce the the terminology here so the X's those are the set of points where we have a known value so these are our X's here that's where we we have those known values that we've measured the wise are the the points that we've actually observed so those are the known known values of our function corresponding to the X's that we have the X Stars here those are the points that we want to make predictions at so in in this case the X stars are basically the entire x-axis domain we've just very finally gone through and said like we have 200 x stars in this in this domain here okay and so those are the the points that we want to predict our function at and then lastly the Y hats are the actual predicted values that we're making so the the darkest one is showing the mean and then those those lighter samples are are just showing different samples from the distribution okay so that's that's sort of breaking it down and now we'll we'll we'll build up what what we're actually talking about here so just like before we can sort of write the joint distribution here where y hat is the predicted uh the predicted values and Y is the values we've actually seen so far so we have our mean function we can get the mean at both the points that we've observed and the points where we're making predictions and then we're just building this covariance Matrix which if you think back to that initial slide on gaussian uh on multivariate gaussians these are just our a b and c matrices that we're now using in our in our gaussian process okay so we can extend this now so this is just the expression for our mean so our our mean Vector here is just a collection of the mean on the points that we're looking at and the The covariance Matrix we've already seen here okay so just like we had the gaussian update equations this might look you know a little intimidating you're like whoa what's that all that is is the gaussian update equations so that same mu of a given B that we looked at it's just it's that's all this is so we're just looking at uh we want to make this prediction the what we want to predict at y hat given our y values that we've already observed and then we have just this is just from the the gaussian update equations here so this is the mean at the at the all of the predicted values and then we're looking at the covariance between the points we want to predict at and the points we've already seen and then the the covariance between the points that we've we've seen as well and an important point to notice here is that the covariance does not depend on the actual values that you see so you can see in the in the covariance term y does not show up at all so your your uncertainty regions around your your gaussian process they don't depend on what you've seen at all so that's that's just an important point to keep in mind it's only that spatial relationship of where you have data and where you don't have data that's being represented there okay and so we we have some notebooks that we'll see uh later of how you would actually you know code this up and uh implement it so um we'll we'll get to the actual implementation side as well of this all right so this is just another sort of visual example here of these are your your uh fit points those are oops those are the points that you've taken uh measurements at and you've observed the function and then you have your predicted mean here using the the gaussian update equations we just saw on the previous slide and the important point to notice here is that the further that we get from our data regions so where we've actually observed these points the further we get from that the larger our uncertainty is so our uncertainty grows the further away we get from the the data and this is the nice feature about the gaussian process is it sort of tells you hey you're in a regime over here where you don't have where you don't have a lot of data so you shouldn't be too confident in those predictions so it's it's sort of that flag of this is what I this is what I think but you're not very certain about this so you know don't go making any life and death decisions based on this prediction here whereas when you're closer to the points you've seen so far you have you have that lower uncertainty there okay and we also already saw we saw the visual example but incorporating noisy measurements is quite easy so in this in this case we have perfect function uh function evaluation so we're observing it perfectly but in the real world that's often not the case so if we want to incorporate that uncertainty in our in our measurements that we have it's pretty straightforward by introducing so this is the actual function f of x and then we're adding some some random uh noise onto that and so if we want to include that now in in uh the model that we're working with here it's pretty straightforward the only change we're making is adding in this this new term here that's all that's all that we've done we're just in incorporating that uncertainty in in the points that we've seen so far and so just like before we can write the gaussian update equations here and the only thing that's changed is just just adding that in there that's the that's the only change that we have okay so it's it's very straightforward to account for the noise and if we want to just look at how this changes it we already saw that initial example but now around the data points that we have we still have some uncertainty there okay so it's it's allowing us to account for that in our model and if we if we want to compare that to before basically all you're seeing is that the uncertainty is collapsed when we have no noise and when we do have noise we just we're still accounting for that as well yeah using this new professor yes yes so that would be sort of you you would characterize your sensor you would know sort of how how uncertain am I in the values that I'm getting and based on that you would sort of put that into your model and say this is the the sensor uncertainty here what we're showing here I believe is the 95 confidence bound so it's like the whatever Z Star score that corresponds to yeah but yeah it can it can vary too okay so we're working our way through probabilistic surrogate models the the next thing we're going to look at is actually fitting the gaussian process so we've got some data and now we want to fit that data the best that we can so I sort of already touched on on this example with the the phone prices but we'll dive into it a little bit more so we have some some data some historical data of phone prices to sales that we have here and we want to our boss comes to us and says I want to uh I want to look at changing the price of this phone and there's three different three different prices I'm considering changing the price to and I want you to tell me what's our our forecast for the new sales that we're going to to get so for this example like let's say just for visualization purposes these are the actual values that we would see we don't know that though all we can see is the Blue Points that's all we've observed so far and so if you hadn't taken this class maybe you would just go to your boss and say well I fit a line to this data and here is what I think the uh the prediction is and so maybe you get lucky and your boss decides to go with you know the far price on the left and you're not that far off from your prediction and you get a promotion but maybe maybe you're unlucky and your boss decides to go with the middle price point and you're way off and your boss is not happy with you at that point so basically we can we can often do better than just the linear fit by by looking at the maximum likelihood of of our data so our goal here is we want to find the parameters so we're going to parameterize our gaussian process and so if we want to think about what what is the parameter here well for the squared exponential kernel that's our length scale but you can even take that further and you can say well the the kernel function itself is the parameter so maybe you have different kernel functions as your parameters that starts getting a bit complicated so for our example we'll just talk about length scale is one of our parameters here okay so we want to we want to look at the probability of our data that's our y given the X's that we've seen and the parameters of our model okay so what we're actually going to work with is the log likelihood here and so for with our with our gaussian you can remember that we just have the expression it's just this uh 2 pi to the negative n over 2 and then we have the inverse or the negative one-half so the square root of the determinant of our Sigma and then we have the the exponent term here okay and so if we want to take the log probability of this all that is is just taking the log so if we do that we take the log of this expression we're just going to end up with uh negative n over two and then similarly we're going to have just the log determinant here and since we're working with logs remember we can just just add them up and then the log here is going to cancel out and so we're just left with that term on the inside okay so pretty pretty straightforward for the for a gaussian distribution we just got something some nice analytical expression like that the good news is that if we want to extend this to the gaussian process we get the exact same thing the only difference that you see is that we've introduced the uh the K of theta so that's our our covariance Matrix now is just instead of instead of Sigma all Sigma is is just the the kernel function parameterized by whatever our parameters are in this case our length scale and then plus that uh that uncertainty term similarly the only difference in our mu is that it's we don't write it as Mu now we're saying it's it's m Theta of X so it's just a mean Vector based on the the data points that we're we're looking at so this is great news because we have this analytical expression for this and so that makes maximum likelihood estimation really easy so the good news we can fit the parameters of the gaussian process using maximum likelihood estimation okay and so say say in this example we go and we actually do that this is just these aren't real values these are just you know for for visualization maybe we go and do that and we get a great looking fit um and we we do really well and our boss is much happier with us than than our linear uh regression estimate and on top of that we can also tell our boss how confident we are in our prediction based on on the modeling assumptions that we've made okay so that's that's how we would go about fitting the gaussian process so just to to summarize the probabilistic surrogate models that we've seen so far just all they are is just a distribution over functions now we've just saw that that example where we just scaled it up they're just distributions over functions and we've seen that the choice of the kernel that you're actually choosing that has a pretty significant impact on the uh the smoothness of the functions and the types of functions that you're getting we also saw that we can we can incorporate measurement noise it's relatively straightforward to do that and then lastly we can fit those parameters using maximum likelihood estimation yes first one um can you represent other types of noise besides additive using aggression process or application yeah that's a good question um what what types of noise are you are you thinking about there for example right where you're like noise is dependent on your particular yeah I I think you should be able to yeah yeah you can't yeah you wouldn't be able to just add it in like we did with the new but it seems like that there probably is a way to do that um and then uh if you're trying to for example extrapolate your fit rather than trying to interpolate between values do the kernel functions need to be kind of bounded or some of them be unbounded as you kind of get further and further away from the points you've actually hit yeah good question so um in the example that we saw here looking at the extrapolation so here because we were just using uh the squared exponential kernel with a zero mean you sort of see that decay it's just going back to zero because we're saying our mean function is zero so as we get further further and further away you just Decay back but if you're using a different mean function um different kernel function I would say still has some impact but the main result you're seeing here is due to the zero mean so if you had some some different type of mean function maybe you're looking at like you're saying it's a linear based on the points you've seen then yeah you could be unbounded and you could just keep increasing based on that uh that relationship that you saw yep okay so we're gonna transition now so we've we've sort of seen how to uh how to model with the the gaussian process now we're going to transition from how to model to how to sample right and that's that's what our our optimization is sort of focused on is selecting these these sample points to to get our function evaluations at so we still want to keep this this high level you know roadmap picture in mind where we we have you know few samples at the start and then we're we're choosing our sample locations so that we better learn what the function actually is that we're interested in and so that's what the surrogate optimization is focused on is is how to sample the function okay so the the first one that we're gonna look at is prediction based exploration and so if you think back to quadratic fit search that's essentially the same exact thing as uh prediction based exploration so as a refresher in quadratic fit search you're just looking at the the three the last three bracketing points and you're fitting a Quadra quadratic function to those those points so that's kind of like your surrogate model right you're just saying okay we have a we have a quadratic surrogate model and then we're going to take the minimum of that quadratic function and we're going to sample wherever that minimum is and that's all prediction based exploration is doing the only difference now is you're not using a quadratic function you're using the mean the predicted mean of your gaussian process okay so it's the same exact thing it's just your your model is no longer the quadratic function okay so if we want to look at look at this in detail we're taking the sample at the minimum of the predicted mean so we can actually look at this through through a real example here so to to break this down the black function just like we've been looking at this black curve here is the true function the data that we've seen so far is this this point here and there's another Point here the red point is the point that we're actually going to take the sample at so there's there's a black point it might be hard to see but there's a black point underneath that red point the blue is the predicted mean from our gaussian process so we look at where is the minimum of this predicted mean and in this case the minimum is right about here so then we say okay we're going to sample at that point just like we did in quadratic fit search so we go and we sample there so we take the sample we've now added it there's two samples here it kind of looks like one but there's two we update based on the gaussian update equations so we update our predicted mean we see that the new predicted minimum is now right about here so then we we say okay we're going to sample there at that red point so we go and Sample at that point and then we repeat and we update our our gaussian process we get a new predicted mean and then we we select another sample location and then it gets us pretty close to the minimum so we can say okay great we're done prediction based exploration close the book we're good to go the only issue with that is that obviously this was a very nice this was a very nice example to start out with so you can imagine and it's not very hard to come up with a counter example of where this this wouldn't work out too well okay so if instead our function had looked something like this right and so we had started over here with these two points and we had said that our predicted mean looked something like this you know something like that and then we went through our sampling it would take us you know to this local minimum but then we've totally missed out on this one over here since we were just focused on what was going on over here and we never even considered this part over here okay so that's the issue with prediction based exploration is that often we we sort of just get stuck in these local Minima and it's not really doing any exploring of the space the other thing to keep in mind is that it's not taking uncertainty into account at all so we just went through all this work of making sure that our our probabilistic surrogate model is representing the uncertainty in our data and now we're not even we're not even using that so it's kind of a kind of a waste like why did we even do that in the first place it does take it into account a little bit when we're actually doing the the mean prediction so it's there but we're not really using that in our in our exploration and then the other point is that the new samples are often very close to the existing samples so it's it's sort of a waste of sampling there all right so if we want to do a little better and maybe now we say okay well we didn't do any exploration at all so let's let's do some exploration here so error based is looking at where the standard deviation the predicted standard deviation is the greatest so we're choosing our samples to maximize that uh that predicted standard deviation and so you can think about this because the the larger the standard deviation is the more the more uncertain we are in the actual uh true function there so we're sort of saying we want to become more certain in our function and that's that's sort of our our guide here for this exploration so if we look at again same exact example here we're going to start out and now we're we're also visualizing the uncertainty here so we start out and instead where prediction base was just taking a sample here this goes directly to the bounds of this of the domain and so because our uncertainty is greatest at the bounds we just sort of clamp those down first right away so we sample at the left bound then we sample at the right bound and then we sort of get into it of uh actually going through through uh and it so the the next highest uncertainty is sort of on that left Peak there so we sample there and now we're sort of just like playing whack-a-mole with the uncertainty right so we we clamp one down and then another another one pops up so we go and we whack that mole and we're just sort of continuing to play whack-a-mole here with with error based exploration and so we can just keep going through it wherever the the uncertainty is the highest sample at those locations and we end up we we end up with a pretty good you know understanding of what our function looks like and we've distributed our samples pretty well throughout the the sampling process the only issue here now is that often the functions that we're interested in are defined over very large domains sometimes just all of RN right and so they're they're unbounded functions and in the case where they are unbounded you saw what What error-based exploration wanted to do it goes straight to the bounds in this case we have bounded it so we put the domains as only it can only go between those so if you're working with problems where you know the bounds pretty well like say for the the phone price example you know that your your phone the the sales corresponding the price the price has to be greater than zero right and you're not going to make it you know a million dollars for a phone so that one we'd say is pretty well bounded but in a lot of other applications you don't have those bounds on it and that's where where you run into some issues with with error based okay so it has to be constrained to that that closed region yes what was your problem is kind of high dimensional would it try to go to all the corners first I guess yeah yeah so if you took it like into many higher Dimensions where you're looking at I don't like you have some hyper Cube that you've bounded it to you're going to go to because the error is going to be greatest unless you already have samples at those those bounds so if you don't have samples there that's where that's where you're going to go and Sample at those those bounds first yes uh what is the relation between Sigma hat and the kernel the sigma hat and the kernel yes that's a great uh great question so you can back out Sigma hat