Optimization 2.0: faster, fasterrr!

Hey guys, Sal. I just released a new optimization mode and it is significantly faster. Not only that, it also gives us new options and features which are super handy. I've been working on this for months, so I cannot wait to show it to you. So, let's get right into it. So, in front of me, I have two terminals open. On the left side, I have the new version of Jessie, which has a new optimization mode. And on the right side, I have the old one. Now, I did this because I want to run the same strategy using the optimization mode so we can see the differences to compare the two together. Now, one way to recognize the new Jessie is using this new welcoming page, which will print out the Jesse logo really big. Now, of course, this is not going to affect your trading results, but I just thought it's really cool to have. But anyways, let's go to the Jesse dashboard. And notice the one with the port 9,0001 is the new Jesse and the one with the port 9,0002 is the old one. So, let's generate a new strategy and we're going to call it optimization benchmark. I will also do the same on the new version of Jesse. Instead of writing a strategy from scratch, I'm going to copy the code for this one from our website. Go back to Jesse and paste it here. I will also do the same thing one more time here. Now, before I forget, we also need to copy the name of the strategy to be this new one. So, let's save it and also do the same one here. All right. So, now we need to update the strategies so that it should be prepared for running the optimization. Now, now instead of preparing a strategy for optimization manually, I'm going to do the lazy way. So, I will just copy it and open JGPT and I will say this one. Prepare my strategy for optimization and I will paste it right here. All right. So, it gave me this one. Let's copy it. Go here and paste it here. All right. So, notice that now instead of hard coding the values for the period of this SMA and this one, we are using the HP which stands for hyperparameters. And here it has defined the hyperparameters function for us which has these two values with the type of int the minimum value being 100 and here being 10 the max being 300 and 100 and here are the default values which are the ones that we had hardcoded in the previous strategy code. Now by the way in this one we are only trying to optimize two integer values but we also have support for float values and in fact with the new optimization mode we also have support for categorical type. Now, if you don't know what that is, that's fine. I will show it to you in the end of the video. All right. So, let's pick the optimization mode from within the dashboard. So, again, this is the new dashboard. And the way I know this is because in the previous one, let's actually open it. We only had one duration, but with the new one, we have durations for training period and the testing period. Now, in the previous version, the training and testing periods were being generated behind the scenes for you. So, you only had control over the total duration of the optimization run. But with the new one, you can specify whichever period you want for training and for testing. Another difference is that you can also specify the CPU course to use right from here instead of from the settings page which was how we used it in the past. So in fact, let's update this one to be another number such as 12. And I will leave all the other options to be as they are. All right. So let's also pick this strategy. It is simply called optimization benchmark. And I'm going to choose the 4 hours as a time frame. the route is correct. As for the duration, I will pick until the end of 2024 and I will leave the number of trades as 50. I will also turn on the fast mode. All right, this looks good. Now, let's also go to the new one and pick the same strategy. I will change the time frame to 4 hours. We don't need any data routes. As for training period, I will pick 2024 until the end of it. But for my testing period, actually just to keep them the same, I want to make sure the testing period of the new dashboard is the one we had for the previous one. So in the previous one, we were using 85% of the data for training and 15% for testing. So approximately that makes this like two months less. So again, let's make sure this one is correct. And this one should also be turn 24 since the beginning and until the end of it which in other words would be the beginning of 2025. As for optimal number of trays I will pick the same number 50 and as for the CPU course we are using it is set to 12. The fast mode is already on. All right. So first I'm going to run the optimization using the previous version. So let's click on the start button. All right. All right. So, I'm getting this error because I implemented the hobby parameters method on the strategy of the new optimization mode, but I forgot to do the same for the previous one. So, let's also open this one and paste it here. Go back and run it again. All right. So, it began working. And as you can see in the previous version, we had to run the optimization in two phases. The first phase which this one is in right now, we had to generate the initial population and then use that initial population to evolve it and find good parameters. But with the new one, that's not the case. We just begin optimization and begin evolving immediately. And this is because the algorithm behind them. The previous version was written by me and it was using the genetic algorithm. But the new version is using the Optuna library and their algorithms which have been optimized through the years and they work really fast and really good. All right, so it's been about 45 minutes since I started this and we are at nearly 80% and we cannot see the sharp ratio in the previous dashboard, but we can see the training and testing P&L and the total number of trades and the win rate. All right, so let's cancel this one and go to the new dashboard. And again, notice that everything is the same. So, and using again 12 cores of CPU. Let's run this. All right. So, first of all, it's going so fast that I don't think it's going to take more than just a few minutes. It's been 23 seconds and we're already at 34%. We also have this chart here which shows us the objective progress. We also have the ability to change the metrics value here. By default, it is set to whatever you choose in the settings for the fitness function. Now, by default, it is set to sharp and that's the value here. But I could change it into other things such as calma ratio, omega or even the piano. Also, while this is going, we can see here that in the new table, we have the ability to click on this to get more info. So, we can see the parameters, the translated values of it. We can copy the DNA from here. We can also see the full list of the metrics for both the training and the testing period, which allows us to compare all the values that we actually care about. All right, so it's been 59 seconds and we are at 92%. So that one took nearly an hour, but this one finished in about 1 minute. So that's how much faster this new optimization mode is. And you don't even have to copy the DNA, go to the back test, paste it there, just so you can see the translated values or the other metrics. You can simply click on this button and see all the metrics that you actually care about. And again, you can also see the parameters. Now, if you like the values, just like before, copy the DNA value from here. go to the strategy and add a function called DNA which simply returns this string value. Now notice that the new DNAs are a bit longer than the previous ones, but that's perfectly fine. And also in case you're wondering whether you have to update your existing strategies to work with the new one, well, you don't. I made sure that if you enter a DNA with the previous format, it will continue to work so that you guys don't have to update your strategies immediately. But if you use the new optimization mode and get a DNA with the new format, this one will also work. The reason the new optimization mode is so much faster is because we are using first the optuna library which is this one. And second, for doing multipprocessing, we are using the ray library which is super optimized. But there's a caveat. The ray library at the moment of recording this video is not supported on Python 3.13 which is the latest version of Python that we support Jesse on. So if you want to use the new optimization mode, make sure your Python version is nothing more than 3.12. All right. So now that you can see how much faster the new optimization mode is, let's continue so that I can show you the new options that it supports. So one thing is that previously we had the minimum and the maximum values for let's say an integer type that we wanted to optimize. But with the new version, we also have another option called step. So for example we can specify 10 and now it will not try for example the number 101 102 it will only increase by 10. So for example 110 120 etc. This by the way was one of the most requested features of JC for the optimization mode. The other thing that's new is the support for categorical type. Now let me show you what that is and how you can use it. So for example here in this strategy we are using the SMA indicator. So what if I wasn't sure whether to use SMA or EMA? I could simply come here and say this MA standing for moving average equals HP MA type. Sorry, this should be self.Hp. All right. And I'm going to say if MA equals SMA, then return this. If it equals EMA, then return this. Now, we don't need all the rest. Now, this one is good practice. So maybe I just keep it here, but you don't need it. All right. So if the MA equals SMA, we're going to return TA. SMA, but if it equals EMA, we will return TA. EMA. So let's copy this one. Come down to the hyperparameters. Now here now I have the option to define a new parameter called MA type. And the type of it well, it's not CR, it should be categorical. And the options are SMA, EMA. Now, they could be as many as you like, of course, but we only care about SMA and EMA. And we also define a default value, which is SMA. All right. So, now let's go back to Jesse, pause, and start a new session. And start it again. All right. So, let's click on the info button. And now, if you check out the parameters, you can see we also have the MA type. And for example with these results we are using the EMA. But if I check another one for this one is also EMA. This one is also EMA. This one for example is SMA. You see? So with this new type, you can use the optimization mode to choose the right indicator for your strategy. Now this can be super helpful and I'm going to leave it to you to come up with all sorts of ideas that you can use it with. All right, so let's pause this one. Again, notice how fast it is going. It's crazy. So, we also have other options. For example, how many trials you want to run per each hyperparameter. Well, that one is now a settings. Now, go to the settings page to the optimization tab. And here you can modify the number of trials you want to run per each hyperparameter. So, for example, if you run optimization and you see that it hasn't yet found a good enough parameter for you, you can come here and increase this number. Or for example, if you found that it has already found a good number and it is just working too hard or maybe overfitting your strategy, you can come here and lower this number. Or if you don't care about it, just simply use the default option. As you just saw, the new optimization mode is significantly faster, which allows us to run the optimization mode whenever we want. We don't have to wait for hours or days for it to finish. Instead, it's going to finish in a few minutes or hours at the maximum. Now, I think this changes everything and how we tackle our strategies altogether. I'm super excited to see what you guys build with it. So, make sure to post a comment and let me know about it. Thank you guys for watching and I'll see you in the next one. [Music] [Applause] [Music]