Bayesian Optimization for Hyperparameter Tuning with Scott Clark - #50

Key Takeaways

[Music] hello and welcome to another episode of we'll talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam Charrington this past week the conference finally came to me over the weekend the great and not so little anymore strange loop conference grace downtown st. Louis I got a chance to meet with a bunch of the speakers including Sumi chintala of Facebook Alison Parrish of NYU and Sam Ricci of stripe I had a ton of fun and I can't wait to share some of these great interviews from the conference before we move on to the show speaking of conferences we're going into conference giveaway mode for a few days next week October 10th through 11th I'll be in Montreal for the rework deep learning summit and one lucky listener will get a chance to join me there entering the contest is simple just head on over to Twilio comm / DL summit and choose any of up to four methods of entry and voila while there are only four ways to enter this time around by sharing the contest with friends each participant can get up to 14 entries this giveaway will only be open until noon Central Time on Wednesday the 4th so make sure you get your entries and ASAP good luck this week I'd like to introduce a new sponsor NEX osis and thank them for sponsoring this week's show next SOSUS is a company of developers focus on providing easy access to machine learning the Nexus machine learning API meets developers where they're at regardless of their mastery of data science so they can start coding up predictive applications today and their preferred programming language it's as simple as loading your data and selecting the type of problem you want to solve there are automated platform trains and selects the best model fit for your data and then outputs predictions get your free API key and discover how to start leveraging scene learning in your next project at Nexus comm slash twimble that NEX OS is calm slash Twi ml head on over check them out and be sure to let them know who sent you finally before we dive into the show a reminder about the upcoming twill Moe online meetup on Wednesday October 18th at 3:00 p.m. Pacific time we'll discuss the paper visual attribute transfer through deep image analogy by Jing Liao and others from Microsoft Research the discussion will be led by Duncan stutters thanks Duncan to join the meetup or to catch up on what you missed from the first two meetups visit Twilio comm slash Meetup as you all know a few weeks ago I spent some time in San Francisco at the artificial intelligence conference by O'Reilly and Intel nirvana while I was there I had just enough time to sneak away and catch up with Scott Clark co-founder and CEO of cig opt a company whose software is focused on automatically tuning your models parameters through Bayesian optimization we dive pretty deeply into how they do that through the course of this discussion I had a great time and learned a ton but be forewarned this is definitely a nerd alert show and so without further ado on to the show alright hey everyone I am here with Scott Clark Scott is the founder and CEO of a company called cig opt and he was gracious enough to spend some time with me this morning to talk about his background the company and the topic that I am very interested in learning more about Bayesian optimization we're sitting in his in his office in San Francisco I happen to be in town for the AI conference and I'm really looking forward to this interview so welcome Scott thank you so much really looking forward to it as well awesome awesome so let's just jump right in and have you tell us a little bit about your background and how you got involved in machine learning definitely so I first got really excited about this while I was in grad school so as a pursuing a PhD in applied math at Cornell University go upstate New York yeah exactly Cornell is great because there's not a lot to do and it's super bad weather all the time so you just focus on studying and he graduate as soon as possible I went to RPI undergrad also upstate New York and had the same experience yes I highly recommend it for efficient degrees RPI had the added advantage of it was hugely skewed towards male students and so they were even less distractions fair enough that's excellent so basically I was applying math to a variety of different things one of the focuses of my degree was bioinformatics so I was my I had a fellowship from the Department of Energy so the problem I was trying to attack was genome assembly and you can think of this as trying to solve a jigsaw puzzle on a supercomputer so basically we have a bunch of DNA and we have to reassemble it into some genome and the Department of Energy cares about this because if you know the genome it might be a path towards more efficient biofuels or something like that okay the problem was lots of tunable knobs and levers with these various systems and we had to configure those to get the best possible performance out of them and you are the grunt in grad school exactly tune all these lovers we joking they call this graduate student descent the idea being we just need to get to the best configuration and it does matter how you get there yeah and so the standard way to attack this problem and a lot of get your feel value for your paper and something like that yeah I mean academic incentives are a completely different topic yeah I just thought I'd top that but the idea is there's a couple standard ways people go about attacking a problem like this you could try to brute force the problem so just lay down a grid of all possible options for every configuration and try them all right this was intractable for us because it took 24 hours on a government supercomputer every single time we wanted to try a single configuration randomized search has become very popular especially in the deep learning literature for trying to come up with different configurations of hyper parameters and architectures and things like that yeah turns out much more efficient than grid search but this is still like trying to climb a mountain by jumping out of an airplane and hoping you land at the peak not necessarily the most intuitive way to go about optimizing something a lot of the different algorithms will use randomized initialization that's different from randomized search correct so when you're building a neural network you might use randomized initialization on the individual weights and then use some sort of stochastic gradient descent optimizer within that underlying system this is more of a black box parameter optimization problem I'm talking about what we're not introspecting the underlying model but just tuning the higher level configuration parameters so some of those configuration parameters might have to do with that random initialization or the stochastic gradient descent parameters or something like that you definitely need to be able to bootstrap efficiently from no data but doing purely randomized search is not necessarily the most efficient thing you can do so maybe before we move on since I think we're going to be spending a lot of time talking about hyper parameter optimization in there you know let's maybe dig into grid search a little bit more so that we're all starting from the same place basically as I understand it the idea is you've got some set of hyper parameters you know those form and an n-dimensional you know n dimensional not a cube but a lattice thank you and you know grid search is basically systematically you know going from point to point like if you were searching for someone in a forest you'd kind of form a grid and kind of attack all those points yeah where and random is you're basically picking points and the idea is statistically if you pick enough points you'll get some level of coverage of you know all of the combinations of these hyper parameters exactly so back to your searching in a forest analogy this is yet jumping out of a helicopter and seeing if the person's there writing back again and continuing to do that over and over again that's random search that's right to my search another popular method is just manual tuning so trying to do this in your head and in the forest example when there's only two dimensions you might have a lot of intuition about maybe the person is going to be up on a hill or something like that it can actually be somewhat effective but once you start to look at 20 dimensional problems a lot of human intuition starts to break down and you might not be able to have some of that expert knowledge in that the searching for a human and a forest setting how to set stochastic gradient descent parameters and number of hidden layers and learning rates and all these sorts of things it starts to get very convoluted very quickly and so manual search well it can be effective to kind of resolve very localized solutions is not a great global optimization strategy and for the typical model that you are seeing like how many hyper parameters are there yeah so it really depends on the underlying system so something simple like a random forest might only have a couple that you care about number of trees number of samples needed to split a node something like that as you start to advance maybe integrating boosting methods and you have learning rates and other sorts of parameters you can tune but once you get into the deep learning and reinforcement learning regimes there can be dozens of individual parameters especially if you've started to think of the system as a whole so when you're doing an NLP or computer vision type problem all sudden you have different ways you can parameterize the data as well and so by looking at that system in its entirety all of a sudden there can be dozens of parameters and something that grows exponentially like a grid search is completely intractable the human manual intuition starts to break down and randomized search is just too slow to lock into a reasonable solution okay can you give an example of in the case of NLP how the the way you look at the data set changes and increases your a parameter space yeah so how you tokenize the text itself so do you look at different engrams sizes the idea being do you look at one word at a time pairs of words triples words do you maybe do different thresholds for the frequency within the corpus itself so maybe cut out words like uh because they're too common and then also cut out words like Bonanza because they're too rare right and so you can kind of change the actual feature representation itself before you even feed it into the machine learning algorithm but these are all tunable knobs and levers got it okay and so you were stuck in grad school like again twiddling these these levers and you know as all innovation happened you thought there's got to be a better way exactly so when around the department and found that this was a very common problem people in machine learning people and financial engineering like everybody were building these these expert systems but they needed to be fine-tuned but everybody was using these kind of standard techniques so expanded my search outside the department and eventually found who would become my PhD advisor and the operations research field so they've been attacking this problem for decades if you have a time-consuming and expensive to sample system how do you most efficiently get to the best configuration so this crops up if you're tuning a particle accelerator it crops up if you're trying to decide where to place a goldmine which is where some of the original research came from in the 50s but it Maps extremely well on to a wide variety of computational problems okay you have some input that comes in some output that you care about how do you get to the best output and as few input attempts as possible so I started working in this field of optimal learning is called an Operations research or sequential model based optimization or Bayesian optimization a lot of fields have different names for it but the idea is how do you do this as efficiently as you can ended up pivoting my PhD towards working on this problem it ended up being one of the the chapters of my thesis and after graduating I realized that a lot of different people in a lot of different industries had this issue so I spent two and a half years at Yelp working on their advertising applying these same techniques to help do more performant advertising okay the idea being if you think about it mathematically an advertising system is very similar to a genome assembly system insofar as a lot of experts spend a lot of time building something there's a bunch of inputs and there's an output you care about an genome assembly it's better papers because you get a better genome in advertising system a bunch of money comes out the other end I mean clearly there are tons of problems of fitness in general exactly exactly and so you started sick up how long have you been at it here yeah so immediately after Yelp started to take up about three years ago went through Y Combinator and winter 15 I've raised a few rounds of funding most recently a series a led by andreessen horowitz and now we're 16 people in San Francisco that sounded like a steamboat I swears that normally those back and so I guess you know I want to kind of jump into you know the the main crux of this interview which is around this Bayesian optimization like walk me through the end of the way and you know folks like Pedro Domingo so talk about like the Bayesian z-- is like this one tribe within machine learning and you know as opposed to others you know kind of walk me through like I guess what I'm trying to get at is like I've had a couple of conversations with folks about you know different aspects of you know like Bayesian program learning and other things but you know I feel like you know there's still some you know they're still like the some ethos of like what it means to be kind of Bayesian and think about things from that perspective that we haven't fully captured on the podcast so if we can like start there you may get to the optimization that would be yes definitely so the way a lot of those other techniques work like grid search or random search is there's no learning happening and I think that's one of the major differences between kind of the Bayesian optimization approach or the Bayesian approach to this problem and some of those more traditional techniques the idea being every single time I evaluate this underlying machine learning pipeline or whatever it is it's extremely time consuming and expensive and I want to be able to leverage that data to decide what to do next and so a lot of the Bayesian methods rely on this concept of trading off exploration versus exploitation so we want to be able to learn as much as we can about that underlying response surface how it varies how all the parameters interact over what length scales how certain we are about specific configurations and how well dope referral they'll perform and learn about that while also exploiting localized information to drive you to better results and by constantly trading off these two facets were able to exponentially faster than something like an exhaustive grid search arrive at better solutions and the main difference here is the fact that we're learning from the past and using that to influence what we do in the future and now I think about this kind of explore explore exploit trade-off one of the things that jumps to mind for me is reinforcement learning does that come into play here or maybe less so because the environment itself the problem itself doesn't you know necessarily change in response to the inputs so the the underlying system can change pretty dramatically so you can think of this as this larger system that that fits around in the underlying pipeline that could be a reinforcement learning pipeline it could be just a standard deep learning or it could be something as simple as a logistic regression or a random forest and you can think about the fact that every single time we try a new configuration we want to observe some sort of output at the end that the user defines it could be something simple like accuracy could be the the Sharpe ratio of a back test of an algorithmic trading strategy or whatever it may be and so we use that to kind of influence what we do next you can think of this as this kind of reinforcement loop as a whole over that entire system but we're agnostic to what the underlying method is I I get that in and so the underlying method is you know could be reinforcement learning or any number of other things but it also sounds I was I guess what I was asking was he are you or could you do reinforcement learning at the top level yeah to optimize the thing that you're optimizing which could be reinforcement learning as well reinforcement learning on the hyper parameter space as opposed to the actual model itself yeah definitely and there's a lot of different approaches to this underlying problem there's a lot of very cool papers that are all the top machine learning conferences for attacking this the way that we attack it is via this concept of sequential model based optimization and this is a very Bayesian approach and the idea is we're sequentially learning as much as we can about this underlying system so once again using the history to decide what to do in the future its model-based in the sense that we're building up different surrogate models for how we think individual configurations are going to respond when we actually sample the underlying system we can use various different things here like Goucher processes or other kind of Bayesian regression type systems and we want to be able to say given what we think is going to happen how do we sample as efficiently as possible so then we want to say what do we think is going to improve an expectation the most what's the highest probability of improvement in terms of that new configuration to suggest and then that loops back into the underlying system after you sample it and we learn update the posterior of these individual surrogate methods optimize on them and repeat that entire process so how do you get to the kind of this proposed model for the model-based piece of this mmm-hmm in general in Bayesian optimization usually you pick a specific type of model and go from there so then some of the open source work I did at Yelp it was kind of very cut and dry music Gaussian process use expected improvement to optimize and go through kind of extremely sequentially this is very similar to Spearmint another popular library let's say that one again spearmint it was an open source library out of Harvard very similar to the metric optimization engine or Bo which I wrote at Yelp also similar like GPI opt which is a kind of a more recent one this is kind of the the bread and butter Bayesian optimization approach couching process is expected improvement what Sukup represents though is this ensemble based approach so different surrogate models different acquisition functions different covariance kernels for learning how the parameters as well as not just kind of that standard build us a single sequential surrogate model-based approach but really taking all of these different optimizers and optimizing it and making it automatic so you can select something ahead of time because you know you want to take a very specific approach or you can take the more generalized approach and say we're not necessarily gonna say we're going to use this specific surrogate model we want to learn along the way what's the best possible thing for that underlying system that we're optimizing right so to take a step back you are in the former case where you're picking a model a specific model you know let's say we're assuming a Gaussian distribution then basically we've got this hyper parameter space we are I'm trying to get at like how you know so the parameters of your Gaussian distribution would be your mean and your standard deviation and how are we like what's the process for for identifying those that is then you know that we're doing sequentially gotcha so the way the daguerreian process works is that it's assuming that the response of that underlying system that we're sampling is going to be Gaussian distributed at any given point so it's not a single Gaussian distribution or something similar to like a Gaussian mixture model what it actually is is an infinite number of potential Gaussian responses for every potential input and then the way the Gaussian processes are analytically defined once you start to sample underlying points you can explicitly build up what that distribution is at sample points or on sample points the main thing that controls this is what's called a covariance kernel and what that is is how much information do I get from sampling point a about some other point B so does it decay exponentially is there some sort of high variance or noise associated with it what are the length scales over which all the different parameters interact this becomes doubly complicated once you start to look at heterogeneous configuration spaces with integers and continuous variables and categorical variables and things like that is this covariance matrix is this something that you're learning as part of the process it's not something that you know a priori exactly so you can set it hyper Airy but you you can also learn as you go so there are tunable parameters around these covariance cards and so it's Turtles all the way down but the idea here is once you can analytically define this is maybe a surrogate function I may use I may use a Gaussian process here's a specific class of covariance kernels like an ard kernel or something like that then you can explicitly say okay how good is the fit given what I've observed so far and because you're defining the system analytically and you've effectively mapped the problem for up from this extremely sparse time consuming expensive underlying system that you're sampling and now you've mapped it over to the surrogate space you can start to throw kind of the kitchen sink of mathematics at the problem and use that to kind of optimize the underlying covariance kernels pick the correct ones find the right surrogate functions and then ultimately leverage that information to decide what's the the point that has the highest probability of improvement or expected improvement or whatever it may be so is the surrogate space in this case the covariance kernel or the kind of this vector this infinite vector of the distributions so the covariance kernel defines that infinite vector or at that functional distribution so there's two ways to think about Gaussian process so your covariance kernel is infinite by infinite dimensions or something on that order or I mean it can or how do you as part of the goal to kind of constrain the dimensionality of this covariance kernel so the covariance kernel itself will take in inputs in the configuration space and basically say okay how much covariance can I expect between these two so it does map into a real number technically for various types of covariance kernels there are these tunable parameters that are continuous so like technically yes there's an infinite number of different ways you can parameterize that right but what we're able to do is say given what we've observed so far what's the most likely parameterization or what's a distribution of likely parameterizations and leverage that to decide okay this is what we think is a reasonable surrogate function and then once again do that across a wide variety of them okay I'm still not fully getting the where the infinite distributions come in yes so there's two ways to think about a Gaussian process one is from the point-wise perspective and so the idea is at every single point we're gonna assume the response from this underlying system that we're sampling it's going to be Gaussian distribution but every single potential configuration has a different potential gaussian response to it so there's some mean and the thing so you've got an input point and then you've got the space of configurations and each of those configurations translates this input point to a different distribution so the the input point is a potential configuration so what maybe I'll take a step back and do it expensive example here so let's say we're tuning some neural network and we want to find the optimal learning rate so maybe initially we try something like 0.5 or something like that we get a response back okay and we're optimizing for the accuracy of a fraud detection pipeline and so we'd be like okay we get 0.7 cross validated AUC that looks alright so the thing that we're optimizing for is our learning rate and the input is you know we're not talking about inputs to our neural network and outputs or neural network we're talking about in aggregate the error well so the inputs are we're gonna be tuning this machine learning pipeline and so at this high like meta optimization layer we're gonna be saying okay we're gonna put in a learning rate and then we're gonna go through the training and cross-validation and all sorts of things and come up with some metric that we care about so maybe cross validated AUC right and our goal is to find the learning rate that Tunes this entire pipeline in such a way that it maximizes that output and so the way that this works in the sequential model based optimization framework is okay so we sampled 0.5 learning right got 0.7 out as as the result and maybe there's a little bit of uncertainty associated with that so then let's say we want to model what we think is gonna happen if we try point six so we have a little bit of information because we've already sampled point five so what we do is we build up this Gaussian process that says okay I'm pretty sure that it's gonna pass near this point that I've already sampled but then maybe the information decays pretty rapidly so I expect to see maybe 0.6 plus or minus 0.1 if I were to sample a point further away from and what you can think of is every potential input learning rate to tune this pipeline has its own Gaussian response that we're expecting it has its own mean it has its own variance and so we can explicitly build that up once we define the covariance kernel and of course as you expand this out into more dimensions so in this example we're talking about what is the covariance kernel look like yeah so we would explicitly set a covariance kernel like an ard kernel that says okay we're expecting some sort of like squared exponential decay of this information from sampling these different points and so is the covariance kernel again in this particular case it's gonna be it's gonna describe the relationship between the learning rate and the output so it's going to describe the relationship between like individual samples of that learning rate so does that vary where we expect wildly different results after 0.01 increments or is it point 1 increments do we expect to be an extremely noisy response although we expect it to be fairly well behaved there's various different parameters of this covariance kernel that basically say how much information effectively do I get after sampling point a about some other point B is the dimensionality of the covariance kernel fixed when we start or does it increase in dimensionality as we sample so it takes in the input which is the actual configurations so in this case it would just be a one-dimensional just the learning rate but you can imagine us extending this out so it takes in a vector which is a specific configuration or two vectors actually and says okay how much covariance is there between these two points these two potential configurations that being said you can parameterize that covariance kernel in different ways depending on which specific kernel you've picked so in something like an a or d kernel which is the squared exponential drop-off there's various length scales that you can tune so maybe we know how chopped-off is that kind yeah does it vary over 0.1 but then something like the number of hidden layers might vary over orders of magnitude larger so like a hundred hidden layers is very similar to one hundred and one but very different than two hundred mm-hmm I'm still not sure that I'm very clear on the the kernel in this specific example right the dimensionality of the kernel is one by one like a scalar it takes in a single value so that's just the learning rate all right well so don't think of it as a I'm thinking of it as a matrix is it a function or is it something out should I not be thinking of it so you can define it as a matrix or it's every point the the pairwise covariance of every point you've sampled so far right so as you sample the dimensionality of this thing is growing of the underlying covariance matrix but the underlying covariance function is just a function so there's no kind of dimensionality associated with it okay so it's basically if I've sampled ten different points then I could have a 10 by 10 matrix which is the covariance matrix where every single actual instance inside that matrix is how does 0.7 Co vary with 0.3 or whatever it may be and this as a whole helps us define the Gaussian process which then gives us this sack astok surrogate function for what we think is going to happen if we sample outside of the points that we've already explicitly observed okay and it does that by way of defining the colonel so how do we get from the colonel some of the matrix to the colonel is the way around so you start with a colonel and then the colonel defines the matrix so every single individual value within that matrix is defined as I got it so we're specifying the colonel in this case you said ADR is aired ard so it's the what is ard stand for I'm said that liking on that all of a sudden but it's the squared Gaussian fall-off yeah yeah so what's now unclear for me is if you've picked a sample in your input space and you've run your your underlying process and you have an output value from that sample is the covariance kernel used to build up like what you expect it to see and then you push that all through and you get what you actually saw and then you can update the covariance colonel and then that covariance matrix gets one more row and one more column because now we have how this new point varies with all of the previously observed points and then we can use that to update our Gaussian process and now we have this new posterior result that we can use to decide what we sample next and what we're doing is we're not just kind of doing naive optimization on that Gaussian process response itself we don't just want to find the point with highest mean or something like that what we want to do is apply an acquisition function to it and say given this is what I think is going to happen if I sample any of these potential input points how do I find the point with the highest expected improvement or the highest probability of improvement or which one's going to give me the most knowledge about the eventual optimize the knowledge gradient method and so acquisition function is a new term that you just introduced is that something that is model-based like the covariance kernels model based on this ad already you pick a model that you use for your acquisition function as well yeah so this is the optimization part of so the sequential part of sequential model based optimization is leveraging the history to build up these surrogate models the covariance kernels keeping it updated and all that's the model based part is actually deciding okay this is what we think the response is gonna be in these unsampled configurations so that's the Gaussian process then the optimization component is given that surrogate model what do we actually optimize for sampling next before we repeat this entire process and so that particular piece is really focused on you know you've got this massive potential state space for your hyper parameters you know how do we how do we choose a sample path through the hyper parameter space that minimizes basically wasting time and not adding information to exactly and this is what really controls that Explorer exploit trade-off right so a popular acquisition function is expected improvement and that is basically how much do I think I'm gonna beat the best thing I've seen so far by so if I've seen a pretty good you see in my fraud detection pipeline now all the sudden I want to be able to do is as well as possible beyond that so we're playing king of the hill effectively another popular one that's kind of maybe a little bit more intuitive to it to grasp is probability of improvement if I were to sample this unsampled point what's the probability that I beat the best thing I've seen so far and so these have different exploration exploitation trade-offs insofar as probability of improvement might be a little bit more conservative like we're gonna kind of keep edging it up slowly whereas expected improvement kind of takes the magnitude of the gain into account so it might try something far away because that thinks there could be something great that it has just never seen before yeah yeah and are there other common examples yeah so another one unfortunately they get a little bit more complicated to internalise but another popular one is knowledge gradients this is what my PhD advisor worked on during his PhD the idea is I'm imagining from the name like that's kind of based on information theory and like how much we're gonna learn by checking this point exactly and the goal is to learn as much as we can about that eventual best point right and so if there's more information theoretic acquisition function and then you can kind of define anything that you want with the goal of event really getting to this best one so these are probably the three most popular but you could imagine doing composites of this or some sort of like upper confidence bound based acquisition function and the idea is you want it to as efficiently as possible trade-off exploration and exploitation because learning about that underlying system and how it performs and things like that's important but at the end of the day you just want the best performing model yeah yeah I think Turtles all the way down strikes me as apt like it's you've got you've got high parameters for your model you've got hyper parameters for your pipeline and then you've got hyper parameters for your optimization system yeah and presumably I'm imagining that you are also trying to optimize the hyper parameter hyper parameters at that top layer for your optimization yes them as well and this is exactly why sig opt exists because there's some incredible research out there a lot of members of our team have contributed to the academic research and a lot of the open source out there there's a lot of promise that Bayesian optimization has but unfortunately a lot of expert time is wasted optimizing the optimizer figuring out the best way to tune all of these turtles all the way down and I think that's one of the places where at least the open source that I released the metric optimization engine even though is very popular on github I kind of failed to deliver on that promise because it required an expert to sit and fine-tune all these different things so the goal of a company like Sega is can we optimize the optimizer for you and create this automatic ensemble that makes all of these trade-offs so that you as an expert can focus on fraud detection and will focus on black box optimization for you okay and so you know we've described a bunch of different kind of variants in this process are there specific you know invariants for sig up in your process like you know like for example you know basing everything on a Bayesian process that's one way of doing this like is the product based around that and and what other kind of invariants are there in the way you approach this yeah so at the very highest level we're just blackbox optimization so there's inputs to a system there's an output or set of outputs that we want to demise and we're gonna try to come up with the best set of inputs so Bayesian optimization is an extremely efficient way to do this especially when it's time-consuming and expensive to sample that underlying system there's lots of different variants of Bayesian optimization so instead of using like a Gaussian process we could use a Bayesian neural network for the underlying surrogate function instead of using Bayesian optimization we could use a genetic algorithm or particle swarm or simulated annealing or even just a convex gradient based method the idea being stickup takes care of that that optimization of the optimizer and automatically selects the best one for you most of our methods or almost all of our methods are Bayesian in nature but we're not constrained to that necessarily yeah I guess that was the question that I was trying to get at like do you how far do you go do you you know also now or envision a future where because you're providing this blackbox capability you know you may you know do you know the Bayesian optimization but also you know sample or test you know the results that you get from particle swarms and other yeah types of methods definitely so in house we've built this very robust evaluation framework for deciding whether or not specific algorithms fare well in different contexts this is what we use when we integrate a new paper and want to make sure that like with high statistical confidence it actually outperforms what we're currently doing and we use this as kind of our our internal metric for deciding what to do but we're agnostic to the underlying methods we just want the best possible thing for our customers it turns out for the types of problems that we're attacking Bayesian optimization is an incredibly good fit and it's kind of underutilized because it's so difficult to get up and running and optimized but we have and will continue to employ whatever the best method is for the problems that we're attacking and because we define this barrier in this way where it's just blackbox optimization the underlying system is a blackbox to us but we're also a blackbox to our customers and so this allows us to kind of hot-swap in the best possible technique to solve their problem and not be constrained in that way okay cool can you talk a little bit about the model evaluation framework that you built yeah so there's some ICML workshop papers from 2016 that go into quite a bit more detail available on our website but the idea is I've just told you that we have an optimization framework that can solve any kind of underlying black box function right like the first response should be how do I know whether or not it's working so internally we built up the system we're kind of traditionally to publish papers and I'm guilty of doing this is you would come up with some strategy pick three to six of your favorite functions show that you can outperform some specific techniques on those functions publish a paper rinse and repeat so when we built this up internally we took the superset of all of those different functions from the academic literature we took functions that look similar to our customers data we took a bunch of open machine learning data sets and strategies we basically piled them all together so instead of comparing against three or four different response surfaces now we're looking at hundreds or thousands of them in addition to that we wanted to make sure against all of these different open source methods and against all of these other kind of different global optimization strategies that we could very robustly outperform them so what we do and the internal evaluation framework is we independently optimize these hundreds of different pathological and real-world problems many times with sig ups and many times with another method and that other method might be just a new version of sig out and then with high statistical confidence we can say which one got to the best value fastest which one got to the ultimate best result which one was the most robust so it didn't have like in the interquartile ranges are all above a specific value it sounds like to draw an analogy from software engineering you built a regression testing framework for optimizer yes so we do use it for regression testing it's run nightly but it's also a way to basically a B test optimizers azrael right you're not using it to order what extent are you using it to inform model choices or I guess the you know what I'm struggling a little bit with is you know so you've got this you've got this you know this of datasets and functions and things like that and if you were trying to optimize across all of those then you've got a least common denominator kind of problem right or a local maxima or something like that yeah so we do have to be wary that we don't over fit to this data set that's definitely true one thing that we found though is the reason why we built an ensemble based approach so let me just just poke at that like I'm not sure is overfitting the right word for what I'm thinking of is is that you know some of it strikes me is the opposite of overfitting whereas like if I were to just look at I don't really care about all this other data I care about my problem yeah like if you're optimizing for this kind of broad spectrum and I can you know outperform you by just focusing on my problem you know I'd probably do that yeah that makes complete sense I see where you're coming at here so this is why we take this ensemble based approach because it turns out like the most popular approach to in bayesian optimization like Gaussian processes with a Rd kernel with expected improvement actually doesn't do super well in a wide variety of different contexts so by slotting in the right tool for the job we can actually hit all of these different facets of different types of problems extremely well that being said no free lunch theorem in computer science still applies here insofar as if you do have expert knowledge about your underlying system and you build a bespoke optimizer to solve that one specific problem you are gonna outperform a general technique that being said you would have to repeat that for the next problem that you attack and the next one or the next one and so the idea is by having an ensemble of different optimizers we use the right one for specific contexts and then a different one for a different context etc so instead of having like the lowest common denominator like you said just the one size fits all what we're doing is actually putting in the right tool and automatically learning when we trade it off so when you're tuning a gritty and boosted method you're getting the right tool but when you tune in er all Network it's still the same API in-state interface but you're getting the right optimizer at it so what I'm hearing is in response to my question like a little both right like your you've built this model evaluation framework because fundamentally you're not necessarily trying to you know outperform a handcrafted model that 50 PhDs has been five years developing whatever you're trying to build a system that can deliver good performance on you know in general what someone throws at it and so you want to test it against a bunch of you know hey these are things that someone might throw at it and make sure that you get good performance and the way that you do that is under the covers you're not just relying on you know one specific set of choices but you're taking an ensemble approach and your optimizer can swap in and out different decisions to produce a result that that's best that's exactly it because what we find more often than not is that people don't assign 50 PhDs for five years for every single optimization problem they add more often than not they're using grid search random search manual tuning maybe an open source solution maybe they have part of their team part-time working on an internal optimizer or something like that and those are the things that we can vastly outperform if you know it's convex and you have gradient information you have a bunch of expert knowledge like there is specific tools that you can use to get there and this is probably a little heavy-handed to use in that situation but more often than not what we're doing is we're coming and replacing these very exhaustive very expensive very domain expert intensive systems and we can generally outperform those to a high degree and I often like to think of the the tool space in general is like there's you know for many enterprises there's such a huge potential opportunity to apply ml that their ability to staff up you know is far outpaced by the opportunities so at a given staffing level like you've got this choice you can either like you know take only the the biggest opportunity and apply all your resources to that in a very manual way or you can you know utilize tools that allow folks to be more effective and bite off some of these you know some some of it's like the a lot of and I'll talk to folks in to talk about it like we only go after homeruns versus you know base hits right and this sounds like this is a tool for allowing people to you know well both go after homeruns as well as try to increase their hit rate for bases definitely and when we fund with a lot of the firms that we work with is how they differentiate themselves from their competitors is not by blackbox Bayesian optimization it's by creating a great recommendation engine or a great algorithmic trading strategy and if you can hire five more PhDs to work on that core differentiator or free up five PhDs to do that and then just use Sagat to tune it they work very additively and hand in hand we can accelerate that time to market accelerate the results getting to the best performance and all of these different things and I think more and more companies are becoming aware of this and using the right tool for the job why rewrite tensorflow when you can use it why write your own Bayesian optimizer when you can use a best-in-class easy REST API awesome awesome so what's the what's the best way for folks to learn more I'm assuming the website yeah stick up calm or just contact at segou comm if you want to shoot us an email we run a complimentary proof-of-concept pilot like we can throw these peer-reviewed papers at you to prove that we're as good as we say we are but at the end of the day we want to prove it with their underlying models themselves so we can work with any Enterprise in the underlying system cloud agnostic model agnostic it's also free for students so if there are any people at universities or researchers and national labs or whatever it is listening to the podcast say Capcom / edu gets you a free enterprise account I wasted way too much of my PHP on the problem don't want to do that for anybody else and what about for folks that are interested in learning about the theoretical foundations of the work where would you point them are there like from three canonical papers or something like that that they should look for yes so if you go to stick-up comp slash research those all of our papers we also have a Bayesian optimization primer there that kind of goes into more detail about some of the things I said verbally sometimes is a little bit hard to describe your processes and things like that the math is there the references for all those papers as well so that can kind of take you down the rabbit hole of all the different ways that this has been applied historic okay awesome well thanks so much Scott has been great conversation and I've learned a ton excellent thank you so much I really appreciate all right everyone that's our show for today thank you so much for listening and of course for your continued feedback and support for more information on Scott and the topics covered in this episode head on over to twill Malaya comm slash talks last 50 next week on Tuesday and Wednesday October 3rd and 4th I'll be at the Gartner symposium in Orlando where I'll be on a panel on how to get started with AI if you'd like to meet up there please send me a shout the following week I'll be in Montreal for the rework deep learning summit and hope to be joined by at least one lucky listener remember to visit to Malaya comm / DL summit to enter contest ends at noon central on October 4 thanks again for listening and catch you next time [Music]