Portfolio Optimization API - Algorithmic Trading with Python and Quantopian p. 12

Key Takeaways

what's going on everybody Welcome To Part 12 of our algorithmic trading with python and quantopian tutorial Series in the last video we covered uh or the last videos we've been covering uh testing multiple Alpha factors combining them and seeing why we want to combine them in general it just it helps um and it's it's just been very clear through the testing here that it's it's been good to us to to to do this so now what we want to do is let's say you found a great combination I mean honestly this combination isn't that bad I mean we still have some volatility here ideally it like what I'd like to see is is like a nice smooth just up curve that would be great like a nice compounding up curve as smooth as possible this one's not as smooth as I would like it but um but this is pretty good and everything's pretty much been improved um so anyway let's say we're happy with this and we want to we want to move to back test it so um so yeah so what we're going to do basically is just take this this this Pipeline and use it because we've got shorts and Longs so we can actually pass this in so um just assuming we were to take this strategy uh we could just actually I'll have to copy that in a second let's just go to algorithms uh and this was this is I think the one that we left on right right so let's just copy this uh come down here new algorithm uh Q tutorial combined Alpha create the algorithm fantastic paste and then we'll come over here take this pipeline copy hopefully that that went well I saw some some weird thing but okay uh make pipeline initialize make pipeline so do this paste now um initially we have this I am going to comment this out though soon enough because we did want to see how we would do but first we just kind of want to see here so we've got the quantiles we've got shorts and and Longs we come down here and we just perfect 50/50 weight on the shorts and Longs uh and we can back test this like now so let me fix the dates here it should be uh we should go with 060 or actually what were we doing we were doing uh 011 2015 and then 011 2016 so of course we're still using a market neutral year over a long period of time the market is not neutral therefore we need to still solve for that problem but one thing at a time young Padawan next run the full back test uh we probably should have changed this to a million but uh oh first of all we need to make our new Imports back to the algorithm uh so what was that import and is it the same I wasn't prepared I think we can just do pipeline. dat I think we can do the same import I don't think that one changes so paste let's just build it real quick and see if it it compiles and then if it does uh I'll continue loading pipeline data if we see like 1% I'll I'll pull the trigger just waiting to make sure we don't get an error now while we wait I just want to in uh introduce the the optimize API okay cool I wanted to talk about the optimize API while the back test is running so now let's change this is this which should be 1 million let's make it 10 million 10 million is a suggestion by the uh uh for the quantopian um allocation so we'll do 10 million run the full back test now do note we still don't pay full commissions we'll we'll check we we're going to change this a few times before we actually care So eventually I will remove it and if I forget someone can feel free to yell at me but the plan is to to to actually do default commissions so so while that's running let me bring up the optimize API so first of all the optimize API is making use of what it's known as convex optimization so you don't really need to know a whole lot about that but basically if it well for one if you want to learn more about it you can go to my the machine learning tutorial Series where we actually where pretty much we cover that through uh when we talk about the support Vector machine but in this case basically the idea is we can this this line here is pretty good you can specify the desired state of your portfolio in terms of objectives objectives and con strength so your objective what does your portfolio want to maximize or minimize so you know minimize volatility maximize Alpha maximize your sharp ratio constraint something that your portfolio has to adhere to uh leverage one okay so leverage must equal one okay in your Universe we've already kind of been through the universe so he's got some examples here initially let's see it came out initially it came out for the uh just for the research uh but now it is available on the actual algorithms one thing to note is this is highly likely to change or at least improve over time I've um there is some assurance that nothing will be um everything should still be backwards compatible but that might that might change okay uh the last time I put out tutorials on quantopian uh they quickly came out with 2.0 so just as a note this might change over time so if you're getting errors or whatever look look up and and see what you what you need to possibly change uh okay algorithm still running just want to check so what we'll do is first of all you you can go through here and play with it in the actual notebook setting if you want I I think that it makes more sense at least in terms of when you're picking your portfolio to really just just do it in the algorithm itself this is kind of like a it just makes sense there so what I'm going to do is there's an algorithm here I'm going to literally just clone this algorithm and I think we'll leave everything possibly here I'm trying to look through this code [Music] um I'm not basically the thing that will change is just here and here the combined Alpha part and then this will stay the same the sector will stay the same so what this will do hopefully is balance us across sectors as well which should be pretty cool um that's just like a new thing to basically add to your strategy so you're not overweighted into one specific sector so already we're doing pretty well and I mean this is pretty pretty much matching what we thought we were seeing in the the uh Alpha lens where we kind of just kind of do crappy or not crappy but also not great in the first few months and then we kind of picked up um and I'm pretty sure we're already on a much better track than we were initially and again this is using the the the quantiles um and I don't think default Commission uh we'll have I'll have to check that one but I'm pretty sure we're well I guess we don't have to I can just view code yeah so we're using this um but we can look here the transactions um there's a fair amount of trans transactions a day so it might have an impact but we'll we'll check that in a minute um and there's maybe a little bit of volatility here I mean you wouldn't want to see it'd be nice if we didn't have such volatility right here um but otherwise yeah we're doing we're doing pretty good so I'm going to keep keep working on this one so while we're here what I'm going to do is I'm going to we're going to Contin waiting on this um I'm going to come over here now this is the our old pipeline basically and I'm going to go ahead and bring these two lines in pipeline data syntax pipeline data Morning Star copy coming over here to the top we're going to make sure we import those two things um he's already imported morning Stars mstar we could use just star. that's okay though I'm I'm not going to worry about that that overlapping import for now uh and coming down here let me think here what we want to do base universe so he's not even using the Q 1500 in this example I kind of want to just because um we I'd like to keep it as close to what we had before I'm just trying to show you how you can work on that that simple skeleton especially because I I see it as being highly plausible that the optimized API is changed and this exact code won't work but hopefully if there is a change you can go to to the to a source on quantopian take his example and make very simple changes to it uh to use what we're doing okay looking at this right here anyway uh huge improvement over what we had before um already um I I'm pretty sure volatility is relatively close but our sharp ratio is above two which is fantastic tortino is great as we probably expect we did beat the market pretty significantly Max draw down is not you know the same as uh return so that's always good now while we're continuing to modify that other algorithm let's do set default commission so or let's not modify commission so we're using the default trade commissions run that back test and while that is running I'm going to come back over here hopefully this isn't too confusing I'm bouncing around I'm trying to just save as much time for everyone as possible so [Music] um so let me now what I'm going to do is we'll probably need to change these numbers I would probably go with because we want to we want to invest in more companies than that than 1.5% that's a in theory a very small number of companies the quantopian fund wants a larger number so let's try to use a large number Plus in theory we're we're dealing with as we've already seen 522 Securities so hopefully a number around 500 total Securities so 1.5% was not going to allow that so so we'll go here um you know 1.5% would allow much less than even you know 100 so uh okay that's good we've got these new Imports the q500 I kind of just want to like throw an q500 in here uh and I could we could almost uh let let's do built-in us Equity pricing that might make sense to have that there uh and then volume must be greater than zero I mean I guess I guess that sounds good I didn't know that was something we would need but let's go ahead and oh what have I done let's see okay so now and Q Q 1500 us monthly top volume Universe equals oh so this is how he's CH he was choosing based on like volume interesting I see why we were why he's using these now we could go off of that I really want to get as close to what we were using as possible uh and just show the difference of applying what we have so yeah I I'm going to uh you can we can try that or maybe I'll add it to the very end or something or maybe I'll put it in the text based version cuz I want to copy this as close as what the way that we've built it in Alpha lens and cut away as much extra so I'm going to do this take this copy we've got the universe testing factor is what we passed so let's come back over here interesting uh and basically combined Alpha will equal what we've just copied so let's come all the way up here paste let's call this something different so I don't confuse myself so we'll call this um optimize plus Alpha combos save coming down here now Alpha is no longer combined Alpha it should be testing Factor we pass the sector sector not n pipe all this we'll leave I hope it runs let's see if this is done and it is okay so we are we lost a little bit of our returns but that's okay we expected that to be the case uh but what's important is now we're paying this is what what you would have actually well in theory if you paid full like personal Interactive Broker account fees so it's actually really good like we we did fine um uh with this combination so awesome now let's add the optimize API to this to help us kind of pick our strategy and just see if we get any Improvement first though let's make sure it runs cuz we really kind of monkey patched this oh we got to fix the dates uh let's do 0101 2015 0101 2016 build hopefully not too many errors combined Alpha not defined where is that in the screen um wait where let's see line 62 I know we didn't uh ah okay fine let's do this let's say uh combined Alpha just so it's even closer uh there okay build algorithm we made it to 62 though so hopefully we can make it all of the way this back test and generate any LS I asked it to build I thought let's try one more time come on start so optimize is a lot slower um than the other method that we were just using so I'll probably pause while this one runs that way if you want to wait till it goes all the way through great and if you don't you can just kind of uh keep watching so you don't have to run it yourself if you don't want looks like it's starting fantastic I'm going to cancel it run a full back test and are we 10 I think we're 10 million or yeah 10 million let's see yep be nice if there were some commas there okay run the full back test okay so while that's running I don't think the other one's running so hope this is our only running back test and uh and I think probably you know what I'm going to recommend is you should probably go through both this and um and I'll try to post the links in the description uh for this notebook and then also the post for running through the API with the uh algorithms um I highly suggest you go through here and read it but then also just kind of play with the code yourself to kind of get used to it but there's not too many constraints and objectives that we can set right now and I cannot REM I don't think there's any documentation yet uh for uh the optimize API so basically it's kind of like a little bit of trial and error uh for yourself actually you know what I think there is document cuz I I've seen some documentation somewhere let me see let's let's check optimize optimize op I forget where I've seen it I've seen um some of the functions for optimize I I think it exists somewhere I I just can't I just don't know where let's see portfolio optimization H well anyway um there are only like very few kind of objectives and constraints that you can kind of deal with right now at least from my brief kind of poking around um but anyway yeah so even now I'm pretty sure we're doing better so far and let me see here view code um we're checking for yeah okay yes so there there's no definition for any commissions here so therefore we're using default Commissions in this strategy as well so we're going to compare this basically to the other one that we just ran I think that this one that had no commissions correct and is working off of just simply quantiles so as opposed to just kind of arbitrarily slicing the quantiles and thinking nothing more of our strategy instead we're we're we're doing something a little more sophisticated uh and let's see if I can uh go to the algorithms here and in fact let me pause while I I'm going to pause while that back test is running let going me see if I can't find what I was thinking I I've read something on the optimize API that kind of went through the methods that and stuff that we could use so let me see if I can't find that um as things are of course it only took me like two seconds as soon as I paused um so first of all if I forget to post these links you can go to Google and and just type like optimize API quantopian and you'll find both of the threads the on that's now available in algorithms and also the request for feedback um but anyway in the one for the algorithms if you scroll down he's posted an updated version of his notebook you can either view or clone it um come into here and there still isn't any official documentation I don't think but this serves as pretty good documentation and examples and if you come down here it's kind of like an overview of how to use the the API and then here he posts the doc strings so here this as I think where I I was seeing what I was talking about a moment ago um basically all your options and stuff like that uh but for now you really you really can just use the skeleton that he posted and it's probably going to be better than um arbitrarily choosing things especially because now you can at least um take note of like things like sector and stuff like that but I highly encourage you to just kind of play around with it I mostly am just showing you as as a simple example as opposed to just splitting quantiles um let's see where is oh here we are okay so this is optimize plus and then was it this one that had the I'm getting everything confused let's see yeah so this was no commission this was with commission and then this one is the one that just ran now so Alpha 06 and then beta 02 same as that um optimize gave us let's see and let me check let me make sure we're going so this one had better returns um and no commission but the the sharp ratio is significantly lower than the one that was using optimized same with sortino of course um volatility here is a little lower and then same with draw down here much less draw down I mean if you can even tell just looking at it this one has more vol ility this is just less volatility it's just smoother our overall return is less but um the way that we got that return is much more um safe and predictable also it looks like we've made yeah much less trades and O Let's see which one was this let's see this one's our six even this one like actually even though we made a lot we we actually went into the negatives at one point and then this one just never went into the negatives fascinating all right so there's a lot of playing around with this that you could do mostly I just wanted to bring it to your attention that it's a possibility now on quantopian and I'm going to go ahead and wager it's probably the best way to go about it because it it just takes the guesswork and you're much less likely to just pick random things and keep trying until something works uh so yeah um the you might want to get rid like certain things here that you might not agree with necessarily like you don't have to go based on like dollar volume and stuff like that so you could remove move some of those things um where are we let's see or did we not even use those maybe we didn't because I think we just like copy and pasted our strategy over that oh yeah so so probably what would be cool is actually to combine that initial um strategy with this one but yeah I'm just this this one's getting kind of long so I want to cut it off here um anyway uh the last thing that we probably should do is with the optimize plus Alpha combos uh where is that it wasn't this one let's see back test okay we can take this back test now that's our code for the back test we can um and what we're going to do now is we can come back to our uh notebook for research I'm kind of undecided if I want to do a uh a new one or what let's just I guess we could do a new one um and then and just quickly do a quick back test check of this so BT equals get back test paste in that string make sure it works it's taking them a while okay and then bt. create full tear sheet so we already kind of knew these numbers um very good ma draw down actually went pretty well um the other thing you might want to check and actually I already ran it so I'll show it to you is the full um let me pull it up for you guys like o over the entire you know sentex time line uh let's see which one would it be uh probably here let me see I'm pulling up the full back test of this strategy against all you know I think it's from yeah June 5th 2013 all the way up to December 18th 2016 and while that's pulling up um we can keep going through here yeah this this looks so much better than the first one that we had very good so anyway um I'm not sure what happened there uh gr de trading volume cool so we've made huge improvements over just by combining um and actually those Alpha factors were all like weaker than the sentiment Alpha the sentiment was already a weak Alpha factor and then you applied two more weak Alpha factors and you've we've made huge progress um so the next steps here are would be to um check for correlation of these Alpha factors so if two of the alpha factors that we're using are correlated like I forget if it was if it was revenue or if it was the operate operating margin or whatever um one of them was like definit it appeared to be correlated to sentiment um so you'd want to check for that uh and here is the uh all time so for total returns match to draw down uh it's not looking too hot Sharps Not Great sortino Is Not Great and volatility is not great so it's better and also this this was uh with default fees so it' be interesting what would happen if we removed if we basically didn't have those default trading fees and this would probably be much better um but what we're gonna what you would probably want to do at this point is fix the whole like we still kind of have a shorting issue we've improved it quite a bit but our strategy obviously is not prepared because it's so Market neutral and pretty even on long and short um we're we're just not doing well enough long term on our bets our bets aren't good enough like we definitely have Alpha we're able to trade on that Alpha we're able to make a profit um trading on that Alpha but it's not good enough we we definitely need to make some improvements anyway um that's enough for now that's a ton of information to throw at you uh like I said I would check out the optimize API kind of go through the uh you can clone the sample algorithm but definitely go through that notebook I guess I probably got rid of it already is it you now available in algorithms and yeah it's down here I'll try to link to it if I forget someone can remind me and I'll I'll link to it um but work through that and kind of get comfortable with using it and really the best way to learn that is is really trial and error in my opinion so um check goes out and if you if it's just too much for you or whatever that's totally fine you can you can stick with the whole quantiles thing the quantiles were fine uh and we we actually I think this was a quantile one yeah like this one used quantiles and and we did okay uh it's just that using the optimized API you're you're much more likely to get even better like nice nicely and smoothed out returns so anyways um yeah so if you have any questions improvements concerns errors you need help with something let me know below otherwise I will see you in another video