Pipeline - Algorithmic Trading with Python and Quantopian p. 5

Key Takeaways

what's up everybody Welcome to part five of our quantopian tutorials and 17 of the Python for finance Series in the last tutorial we just kind of introduced the research environment showed how you could pull in uh some some imported data from quantopian showed how we can kind of manipulate that data either kind of play around with the blaze expression or get it back into a panda data frame so especially if you're like me I'm I'm really familiar with pandas and I know how to do things in pandas and I'm not fully familiar with like all the things I can do in blaze so that's something I personally have to look more into to learn more about using but it's also super quick and super simple to get it out of a blaze expression so and it might be useful to I'm I'm not even sure the command to get it back into a blaze expression to be honest because I I'm sure it's not do compute again um but I'm sure there's a way to get it back in so that might be a good starting point but anyways not on Blaze right now so we're going to be talking about in this tutorial is the or are the the the basics of the pipeline so when quantopian first came out they didn't have a pipeline and basically you had to use this like I forget what the what it was called but it was basically almost like a SQ Alchemy if it wasn't actually an SQ Alchemy query that would F filter your your Stocks by really really rudimentary things basically things that had to have already been existing in data and if if I recall right you could not filter by something like uh Price to Book divided by Peg or or something like that you wouldn't be able to do that um and you certainly wouldn't be able to like kind of create your own signal and then filter based on that signal it just wasn't possible so at least if I recall right so you basically had to whittle down your companies by something really rudimentary like market cap you had just say okay look I want to consider the top I think you had either 500 compies or 255 companies that you could consider at any one time either way it was very limiting so they introduced pipeline because you still have to kind of Whittle down your companies uh but you don't anymore have to only consider 255 or 500 whatever it was you can consider a much larger amount of companies after they made other changes too I think it was mainly just in in the whole data thing anyway what pipeline does is it gives you a really quick and succinct way to to whittle down data sets and it's going to be a lot quicker than running functions on the data frame itself because what pipeline is going to do for you is it is going to return a data frame and what you know your data sets are going to do like when you import data it's going to it's a data frame or a blaze expression if you're in research but it's a data frame especially in in the uh actual algorithms um where you're doing your back test as far as I if I recall right too anyway uh you you can you can map functions to data frames and use that on your own but the pipeline is way faster I mean it's just way faster it's very unfortunate if you have to do your own thing outside of pipeline luckily you can do basically everything in pipeline but that's the point of pipeline is to basically do all your functions that you're trying to do on your data frame do it in pipeline otherwise you're going to you're not going to be happy because it's going to take a really long time to do it outside of pipeline so so with that let's go ahead and jump into it because um I think the best way probably to learn is to just get into it so first we're going to do from quantopian pipeline import pipeline easy enough now let's go ahead and do like a really simple make pipeline function So Def okay Define make uncore Pipeline and then we're just going to Simply return pipeline so what this is going to do for us is normally in here is where in this function is where we would filter down our companies but you should see there has been no filter right so if for example we run this pipeline so first let's do from um quantopian research research import run pipeline now let's actually run it so we're going to say um result what is happening why am I falling apart here results equals um run uncore Pipeline and then you're going to specify what pipeline are we going to run well we're going to do um make uncore pipeline uh do not forget your brackets here or your you know the opening Clos parentheses you actually want to run it you're not trying to pass the object you know or the function or I don't even know if it's a method or a function let's see I guess it would be I don't know anyway not important run it um but don't forget those otherwise you're going to get a nasty error um and then we're going to pick a start date and an end date so start date we'll just we'll just do the same dates here we're just going to do 2015 you can do anything you want I'm going to do 2015 oh is this not let's see what do we somehow we we converted this this is what happened I don't think I'll lose this but I'm going to copy this just in case needs to be code I don't know how that got switched anyway 2015 05- 05 and then end dat equals and again we'll just we'll do the same day normally you wouldn't do this but um pipeline is going to pull from like thousands and thousands of companies and then when we go to pull pricing information and stuff if we're just on one day it's just going to go a little quicker and for now we're just I'm just trying to show you examples so I'm going to run that cool and and just for the record let me run this without the the brackets just so you see the error that you're going to get cuz a lot of times these errors not not to the fault of quantopian errors in general are just hard so here um function has no attribute to execute plan the only time I've ever seen this error is when I forget to put my my parentheses here so just just as a a warning because this is a really common thing to forget um to put there okay so now that we've done that let's go ahead and um result what do you think result is right it's a panis data frame of course actually let me see let's do dur result so I'm actually not sure because pipeline came from research or run pipeline did so let's do or not dur um type okay it is a data frame just making sure result. head and uh so what we have is a bunch of tickers and if we did uh Len result we would see how many we have in this case it's 8,240 tickers so we haven't obviously we have not filtered this at all so now let's talk about how we might actually filter it and just in case I'm going to add some space here I just don't want to run over my my my face or have my face run over the code rather so now what we're going to do is basically uh we don't really need to import pipeline again the only thing I'm going to import again is sentiment because I'm pretty sure did we pull it out here no we didn't I guess I won't I won't but I'm I'm pretty sure this is going to give us a problem if I don't and then the q500 I think yep we've already imported so I'm going to I'm going to just rebuild the make pipeline function now so Define make Pipeline and if we get an error it'll only help me to show you more errors whoops no doubt um okay make my I want it to go away I don't want to stare at that while I build this function much better make pipeline so I'm used to like when I'm typing in a field for example like let's say like a chat box on something right you press shift enter to not send the message right and go to a new line and you press enter to send the message so that's why here I'll probably keep doing that doing shift enter which runs it on an high python note book anyway I'm an amateur sentiment factor is going to equal now sentiment which we imported. sentiment signal which is a column so remember sentiment comes in at the moment it's it's through a blaze expression but we can we can treat this very much like a panda data frame so sentiment signal is just that column and then we're going to apply a factor in our case that factor is latest really simple Factor now we have factors and we have filters and I think we even have something new with classification maybe or something anyway let me let's just go let's go to the docs and um let's scroll down to pipeline um you've got let's let me see if I can do Factor filter yeah so you've got base classes here you got factors filters and classifiers we're not going to talk about that for now but a factor basically is going to give you a number it's going to give you a number back now a filter is a Boolean right it's going to give you a true or false so and then a classifier I guess will give you a categorical output and that's fairly new and I don't know about bound column but anyways at least to start we're just going to um concern ourselves with factors and filters and just know again a factor will give you a value filter will give you a Boolean expression or a Boolean expression anyway this is our factor that we're going to use okay so now we're going to say what's our universe so our universe of companies that we're willing or we're interested in knowing information on basically so that's going to be the q500 US um and that is also something that's going to require little parentheses and where's my aunt there it is and sentiment um underscore Factor needs to have the attribute of being not null right we don't want to have an nothing there because if it's not there we don't care about it so that's our universe and we're going to use the universe as a kind of screen for our pipeline which I'll show you in a moment so pipe is now going to be equal to a pipeline and it's going to have columns and these columns will be basically whatever you can call them whatever you want but it makes sense to be sentiment sentiment there we go sentiment is going to not equal it's dictionary here sentiment sentiment factor I think the IPython notebook also makes me type worse I'm normally not this bad and then like for example we can say Longs positions that we're going to go long we could say something as simple as um where sentiment factor is greater than or equal to 4 and then we could say shorts shorts are where um sentiment factor factor is less than or equal to -2 as I said before it's a range of -3 to POS 6 where -3 is just as negative as six is positive so -2 is just as negative as four is positive okay now um when we get to Alpha lens I'll I'll briefly explain and when we do the back test why like we can't really test this specifically in Alpha lens at the moment and in hopefully the nearish future we can because Alpha lens is already kind of at least the open source version of alpha lens not the one that's on quantopian already supports what would make this better but also it raises another question as far as whether or not you actually want to use that but um if I forget to to hit that someone remind me but hopefully in the next tutorial when we cover Alpha lens I'll explain more on this so yeah just just wait but in our algorithm what pipeline returns is a data frame it's going to be a data frame of you know the index should be like your tickers it's basically a double index it'll be index by date and tickers and then you've got sentiment and then you'll have Longs and shorts long and short will just be Boolean values uh and then every day of trading you can very quickly get that information and this is a perfect example of like doing it here in this pipeline is really quick this operation will happen very fast whereas if every day you actually did like a map of a function that mapped true or false based on these parameters um that that would take not four to six times longer it would take like 10 plus times longer it takes a very long time to do things outside of pipeline so you really want to do everything you can inside pipeline okay once we've defined our columns we are ready to um add the screen which is going to be the univer universe of companies and Screen universe and we're done I'm just trying to make sure I closed everything off here looks good to me so when we're done we're going to just simply return pipe looks good let's get a result let's say result is equal to run run pipeline what's the um pipeline we want to run here is make pipeline don't forget the parenthesis start underscore and in fact let me just uh let's let's grab all just copy this actually I'm going to change these dates so whatever anyway too late so 2015 let's let's go um 011 end date 2016 011 so the year of 2015 is I think a good year to test against because 2015 overall did like nothing but between that Mark the market goes up down and then back up again to basically neutral uh so it's a good year to kind of test against now overall the stock market doesn't isn't neutral it rises over time so your first place to test I think is great for the year 2015 and then you want to look at you know the whole the whole time frame and also maybe even very specifically um you know maybe around 20072 2008 kind of time when when the market is very bearish and then overall it's just mostly bullish anyway um I should have run this let's run this because I we might actually wind up yeah sure enough we round up with an error so field objects has no attribute latest so I'm unclear why I get this error like why in the code I get this error but the reason I'm getting it is for some reason sentiment has been converted which is weird because we did Blaze compute on Bank of America sentiment not sentiment right I don't know but anyways this probably clears up if I just rerun all these things almost there maybe I don't know where I am continue make pipeline good good good oh do I am I still hitting it on result fascinating I'm not sure why I'm still getting it on result let's see because it's all about the uh sentiment signal field object has no attribute latest let me fine let's go uh L result let me just go here and let me just do um from quantopian do oh here's what the problem is okay I would normally probably edit this out but I'm going to leave it here just because this is probably a valid issue so up here we're doing quantopian interactive. data. synx import sentiment that's a blaze expression down here where we don't want to be using Blaze um we want to use the actual pipeline data I told you I would fix this before I got to it and I didn't sorry so okay from quantopian pipeline. dat. sentex import sentiment s importent there we go so that was the issue so don't forget to get out of the interactive data um and use the the actual pipeline data when you're going to try to stuff it through a pipeline I wonder if you converted it if you did Blaze compute if this would have been acceptable it's a good question actually anyway not important so we got our result let's go ahead and just look at it real quick result. head sure enough there you can see the the the Dual index here date and then all the equities we have yes whether we would long and short this is going to be useless to us until we get to the uh the well I guess it wouldn't NE totally be useless but anyways we're not going to use it until we get to because we can't do the whole bit never mind I'll explain an alpha lens why we can't use that anyway we have the sentiment data and what we're what we're interested in now is to know not I guess it's a good time to talk about it not whether or not not necessarily whether or not you know four and -2 is a good number right um cuz that that's our strategy our strategy is to trade based on you know we're we're going to assume a sentiment of four or above is good and and negative -2 or below is bad that's our strategy what we really want to know is is there a signal just is there a signal in this data in this sentiment data that's our question we want to know first is there a signal and then if there is let's worry about making a strategy that's going to exploit that signal but you know making what we're doing here is first making a like before before we had research environment we're first making our strategy and then we're using the strategy to tell us sort of if there's a signal but that's not that's not the best way to go about it so uh in the next tutorial we are going to be uh taking what we've got here and running it through Alpha lens to hopefully analyze Alpha factors so that's what we're going to be doing in the next tutorial um sorry for my mistakes but hopefully uh my leaving them in here will only help you because seeing some of these errors are like you know these are like these are errors that I've never seen in any other place besides on quantopian so I figure it's probably a good good place to good good thing to leave in so I'm going to keep leaving them in so anyways uh if you have questions comments concerns if I if you've got something you could improve uh do let us know otherwise I will see you in the next tutorial where we talk about Alpha lens