Algorithmic Trading – Machine Learning & Quant Strategies Course with Python

Key Takeaways

in this comprehensive course on algorithmic trading you will learn about three cuttingedge trading strategies to enhance your financial toolkit latchezar teaches this course he is an experienced quantitative researcher and data scientist in the first module you'll explore the unsupervised learning trading strategy utilizing SP 500 stocks data to master features indicators and portfolio optimization next you'll leverage the power of social media with the Twitter sentiment investing strategy ranking NASDAQ stocks based on engagement and evaluating performance against the QQQ return lastly the intraday strategy will introduce you to the gar model combining it with technical indicators to capture both daily and intraday signals for potential lucrative positions hello and welcome to this free code Camp course on algorithmic trading machine learning and Quant strategy with python my name is Lazar and I'll be your instructor toout the course and in this course we are going to develop three big quantitative projects from start to end and the course overview would be the following first we're going to talk about algorithmic trading Basics then we're going to talk about machine learning in trading some obstacles and challenges we may face while using machine learning in trading then we are going to develop the first project which would be an unsup rised learning trading strategy using stocks from S&P 500 the next project would be Twitter sentiment and it would be using data from NASDAQ 100 stocks and the third one would be focusing on one asset it would be an intraday strategy using a gar model to predict the volatility it would use simulated data and after that we are going to have a quick wrap up and with that we'll finish the tutorial but before we continue I would like to mention that this tutorial should not be ConEd as a financial advice it is for educational and entertainment purposes only we are going to develop some Concepts and come up with strategies in the end but it's not a financial advice and you shouldn't make any decisions based on it as well for this course I would assume that you have at least some python knowledge and understanding because down the road will deal with some complex problems and if you're new to python you may get B down nevertheless you learn some very interesting Concepts so stay along and let's get into it okay algorithmic trading Basics so what is algorithmic trading it is trading on predefined set of rules which are combined into a strategy or a system it is developed by a programming language and it is run by the computer it can be used for both manual and automated trading by manual what I mean is that you may have a screener which comes up with a set of stock stocks you want to trade on a given day or you may have a algorithmic strategy which is developed into an alert system and whenever the conditions are triggered you get an alert but you execute it manually on the other hand you may have a completely automated complex system which does a lot of calculation comes up with uh positions and sizing and then executes the trade automatically okay so what is the role of python in algorithmic trading python is the most popular language used in Alor trading quantitative finance and data science and this is mainly due to the vast amount of libraries that are developed in python as well the ease of use of uh python it is mainly used for data pipelines research back testing strategies as well it can be used to automate strategies but python is a slow language and it can be used to automate low complexity systems if you have a really high-end system which is really complicated and it needs to execute trades really quickly you would use Java or C++ for those strategies our gmic trading is a great career opportunity it's a huge industry there are a lot of jobs with hedge funds Banks Prop Shops and I've just checked the average yearly base salary for Quant researcher is around $173,000 and this is not including the yearly bonus it is a great career opportunity and if you're interested into it the main things you need to know are python you need to know how to back test strategies you need to know how to replicate papers and you need to know machine learning in trading if you're interested into it I'll definitely advise you to go for it okay so let's move on and talk a little bit about machine learning in trading and some use cases of machine learning when we talk about supervised learning we can use it for Signal generation through prediction for example we can come up with buy or sell signals on a given stock or a given asset based on predicting the return or the sign of the return of that asset we can as well use it in risk management for example we may use a prediction to determine the position sizing or the weight of a given stock in our portfolio or to predict where exactly should our stop loss be and with unsup learning we can use it to extract insights from the data for example we can discover patterns relationships or structures within the data for example clusters and use it in this way to uh help our decisions what are some of the challenges that we may face while trying to apply machine learning in trading and the first theoretical challenge we may face is the so called reflexivity feedback loop and it is referring to the phenomenon that if we for example have a machine learning model that is uh predicting that a stock is going to go up each Friday and we can form a strategy around to profit from that phenomenon for example we are going to buy each Thursday and then sell on Friday to capture that price increase move if we find this strategy throughout predictions and start trading it with time other Market participants as well are going to find this Market phenomenon and start to exploiting it as well which would cause the price to start going up on Thursday because everybody's buying now on Thursday instead on Friday and then this strategy is going to be Arbitrage away so it's this reflexivity feedback loop which is making predictions quite hard what is most hard while applying machine learning the most hard thing is to predict returns and predict prices the next quite hard thing to do is to predict return signs or the direction of a given asset is it going to to go up or down the next thing is to predict an economic indicator for example it is quite hard to predict nonfarm payrolls or weekly jobless claims and a thing which is not that hard or quite straightforward is to predict the volatility of a given asset furthermore there are some technical challenges like overfitting a model or generalization overfitting is that the model is learned the train data too well and it fails on the test data and generalization is that the model is not performing the same as on the real data as well we may have nonstationarity in our training data and regime shifts which may ruin the the performance of the model and the last thing is that if we have a really complicated model or um neuron Network it is like a black box and we are not able to interpret it correctly what is the usual workflow process in algorithmic trading and machine learning that would be to collect and prepare the the data then develop a hypothesis for a strategy then you have to code the model and train the model and then finally back test the strategy okay guys and some key takeaways from this course so you learn high level Concepts in quantitative Finance as well practical machine learning in trading you develop a project from idea to back test final results however we will not automate or execute any trades we'll just develop a strategy this course is all about developing a strategy from start to end so you see the workflow and this is just a purely research project for educational purposes I repeat it should not be construed as any Financial advice whatsoever and with that we can move to the first project and let's get into it okay so first project is about unsupervised machine learning trading strategy we're going to use data from S&P 5 500 stocks let's talk a little bit about unsupervised learning in trading so it involves machine learning techniques to analyze financial data and discover patterns relationships and structures within this data without predefined labels or Target variable unlike supervised learning where the model is trained to make predictions unsupervised learning focusing on extracting insides from the data and some use cases would be clustering which we are going to use in the first project dimensionality reduction anomaly detection Market regime detection and portfolio optimization in this project what we are going to do is first we're going to download all the prices data for all S&P 500 stocks then we're going to calculate different technical indicators and features for each stock next we're going to aggregate on a monthly level and filter only the top50 most liquid stocks from the S&P 500 for each month next we are going to calculate PL monthly returns for different time Horizons to add up to the features and the next step would be to download the F French factors and calculate rolling Factor betas for each stock as well to add to the feature set and at this point we'll have enough features to fit the model and either make predictions or in our case we're going to fit a c's clustering algorithm an unsupervised learning model and we will use it to group group the stocks into similar assets into clusters and from those clusters then we will be able to for each month select stocks from a given cluster and we are going to analyze the Clusters and select a particular cluster and then for each month we are going to select those stocks within this cluster and form portfolios however these portfolios will be optimized so the weights of the stocks within the portfolio we're going to find them by using the efficient Frontier Max Sharpie ratio isue portfolio weights and then we are going to form the portfolio hold for one month and rebalance at the end of the month and you know form another Max sharp ratio portfolio in the end we'll have the strategy returns for each day and we will be able to compare our portfolio strategy returns to the S&P 500 returns themselves actually not small limitation is that we're going to use the most recent S&P 500 stocks list which means that there may be a survivor buyers in this list this is a huge issue actually in reality you should always uh back test strategies using survivorship free bias data what is survivorship bias it is the condition when a stock which have actually went out of the S&P 500 because it was failing is currently not in the list right so last year for example there was a stock which was failing and going down down down so at some point in December last year they removed it from S&P 500 and they included a new stock so if we have made the optimization last November we could end up with having this stock into our portfolio and actually it affecting our portfolio results but if we use the most recent S&P 500 symbols list this stock would not be there so that's survivorship bias and for the given project we are not going to deal with this survivorship bu so the list we are going to work with has survivorship bus most probably so yeah that is limitation you need to know in the second project we are going to develop a Twitter sentiment based investing strategy we are going to use the NASDAQ 100 stocks and Twitter sentiment data what is sentiment investing this approach focuses on analizing how people feel about certain stocks Industries or the overall Market it assumes that the public sentiment can impact stock prices and for example if many people are positive about a particular company on Twitter it might indicate potential for that company stock to perform well what we are going to do first we're going to load the NASDAQ stocks Twitter sentiment data then we're going to calculate a quantitative feature of the engagement ratio in Twitter for each stock after that we're going to rank all the stocks cross-sectionally for each month and create an equal weight portfolio in the end we're going to compare the return of this portfolio to the NASDAQ itself the second strategy is much smaller than the first strategy there is no machine learning modeling in this strategy but the idea here is to show you how alternative or different data in this case sentiment data can help us to create a Quant feature and then create potential strategy out of it that's the idea of the second project and the third project is about an intraday strategy using a gar model in this one we are going to focus on a single asset and we'll be using simulated data actually we'll have daily data and intraday 5 minute data but what does an intraday strategy means this approach involves buying and selling Financial assets within the same trading day to profit from the shortterm price movements intraday Traders usually use technical analysis real time data and different risk management techniques to make decisions and profit from the strategies what exactly we're going to do in this project first we are going to load the simulated daily data and the simulated 5 minute data then we are going to define a function which would fit in a rolling window a gar model to predict one day ahead volatility of the asset after we have that we'll calculate prediction premium so we'll predict the volatility and we'll calculate the prediction premium and form a daily signal from it after we have uh calculate that then we'll merge the daily data with the intraday data and calculate intraday technical indicators to form intraday signal so we will have a daily signal and then on top an intraday signal so two signals and after we have that we'll generate position entry and hold until the end of the day in the end we're going to calculate the final strategy returns and this is it for the third project the idea is to show you how predicting volatility Works in uh intraday strategies and yeah with that guys we are ready to jump into the first project so let's get into it okay let's start with the first project the unup rised learning trading strategy but before we continue with the coding guys the first step would be for you to pause the video and install all the needed packages for this project so I prepared a small list up here with all the packages we are going to need for the project so those are pandas numai MP lip stats models pandas data reader daytime wi Finance SK learn and buy portfolio op how you can install the packages you can do it the following way you can open on Ana prompt like that right and then just write pip install and the package name pip install pandas for example and do that for each package that's one way another way to do it is through the notebook itself you can uh type the following so pip install and then the package name so pip install pandas you do control shift and run the cell and it will install the package pause the video take your time install all the packages and let's continue okay so the first step would be to download the S&P 500 constituents prices data but before that we'll have to import all the packages we use throughout this project I've already prepared that those are all the packages we've just installed and now we are importing into the Jupiter notebook The Next Step would be to download S&P 500 constituents data to do that we can go to this link on Wikipedia and up here you can see that they have a table containing a S&P 500 component stocks like the symbol the the name of the security sector industry date added as well so quite some data uh so we would like to load this table into our Jupiter notebook how we can do that we just call a pandas read HTML function and let's see what this would return us okay so this is returning a list containing two elements it looks like two data frames so actually we are interested in the first frame yeah exactly we will assign that to an object called S&P 500 and the next step would be to grab this symbol column and extract all the symbols in a list but before that I think we would have to make a little cleaning on some of the symbols because I know that one or two of the symbols contain a DOT and this would give us an error while we download data from wi Finance so we have to actually replace all dots with Dash and that would do the job next step would be to you know just grab list with all the stocks so yeah we can just use symbol. unique to list we will assign that to a object called symbols list and yeah that would be our symbols list of all S&P 500 stocks as we talked uh in the beginning this list of stocks is not Survivor ship bias free so you need to know that it's uh quite some of limitation all right so we would like to download the data up to a few days ago let's define an object which is end dat and we'll use 2023 September 27th and the start date would be P to date time and date it would be exactly 8 years ago so we can just use the end date and we will substract eight years out of it how we can do that we just say pandas to datetime we'll convert to daytime the end date and then we'll just substract pandas date of set and then Supply 365 * 8 all right so now we have the end date which is yeah 27th of September which is a string and the start date would be a time stamp but that's all right because we are going to download data from y finance and this function we use from y Finance download function which takes tickers as an argument so that would be our symbols list then start which would be our start date and end which would be our end date and now this will download all the S&P 500 constituents if we run it yeah it will take some time all right so we've downloaded all the data for the S&P 500 stocks we got it up here so the next step would be to actually yeah we'll comment out everything and have uh this object printed out as you can see up here we have the column and then we have like a multi index column so first we have the adjusted close and then we have the adjusted close for each stock that's not really convenient to work with the whole data frame at the moment we have 2012 rows and 3,8 columns that's really inefficient how we can overcome that we would just use the stock method which now creates a m index the first level would be the date and then for each date we have the corresponding 500 stocks adjusted close close high low so that's much more convenient we have six columns and almost 1 million rows so we have to change the datea frame to be stocked or actually we can move this method right up here after the download directly and yeah now we have our data frame stacked Next Step would be I would always when I have a multi-index I would always want to have labels on both of the index levels so we'll say index. names and we'll assign the new name so date and thicker that will be our new multi-index names as you see they change step here next step would be to fix the columns a little bit I I would like to fix the column names to be not as titles but with a lower letters so that would be DF do columns DF do columns do string lower and that's pretty much it with the downloading and fixing data a little bit before we move to the next step which would be to start calculating technical indicators and features of all those 53 stocks okay so in the second step we can start calculating the features and Technical indicators for each stock we are going to calculate the garm class volatility RSI Ballinger bands ATR macd and dollar volume for each stock let's first start with the garm class volatility what is garm class volatility it is volatility measure usually used in Forex Trading but it works for stocks as well it is approximation to measure the intraday volatility of a given asset and that is the formula right here so what do we do we Define a new column called garmon class V and it would be the following so log from the high minus log from the low so that would be yeah from the low this whole thing is squared and then it is divided by two then from this we substract 2 * log 2us one multiplied by subtraction of log from the adjusted close yeah minus log from the open again the hold think is squared and we just close another bracket and that should be it that should be the gar class volatility yes so we now have calculated the gar class volatility for each stock it is calculated on a given role we don't need to do any fancy calculations for this one the next one is RSI so how do we calculate the RSI on each stock what we're going to do we are going to group Buy on the thicker level so level one the multi- index has level zero and level one which is the thicker level zero is the date level one is the thicker so we grew by on level one then we are selecting the column which would be adjusted close and apply the transform method and within the transform method we just apply a Lambda function which would be so now we have grouped grouped by on each thicker and what do we want to do we want to calculate the RSI to calculate the RSI we are going to use the pandascore TA package which is the package to calculate pretty much all of the needed technical indicator so we use the pandascore TA package and from the pandascore we use the RSI function the side function we have to supply the close price which would be X and the length would be 20 and yeah if we do that you see now we have the RSI for each stock how we can double check our work we select apple and then RSI and then we'll plot it and yeah as you can see the r side goes up and down up and down so we have worked correctly the next indicator we would like to calculate is Ballinger bands and actually we would want to have the lower band the middle band and the upper band but there is one specification for each indicator from now on we would like to normalize and scale the indicator itself so for the Binger BNS will will supply the log from the close price first the function we are going to use is from Panda CA B bands and it is taking the clothes just for presentational purposes we Supply a upper adjusted close and the length will be 20 when you run this function it returns five columns so the first one is the lower band the second one is the middle band and the third one is the upper band we have to take this into account and what we are going to do is pretty much you know Define a new column BBL low Ballinger band low and we use the the same idea as for the RSI we are going to group by each thicker select the adjusted close column and then we use the transform method Lambda function and in the Lambda function we say pandas ta B bands close would be equal to X actually it will be equal to log of x so MP log 1 p and then the length would be again 20 when we run that it will return those five columns what we actually want to do is assign to B below the First Column which we know is the B the lower Ballinger band we can say iock all the rows and the First Column we can repeat the same operation for the midband and for the upper band okay but this happens when I forget something and I think I have forgot I forgot the second curly bracket to close the curly brackets let's see what this would return yeah guys after you return and calculate something you can just select all the the code you've used and comment it out okay we forgot to change the names that would be BB mid and first index that would be BB High the second index okay let's run that again and see what we get awesome now we have the lower bager Bond Middle Ballinger Bond and upper Ballinger Bond and we have the data scaled and normalized next step is to calculate the ATR for each stock however the ATR function needs three inputs so three columns not only one column and when we use transform method in pandas it is actually working when you select only one column it would not work if you have three columns as an input so we would have to use another approach more specifically that would be a group by apply and to do that we need to Define our own custom function to calculate the ATR we can uh double check what the ATR function from pandas ta requires as input and it is the high so we have to supply the high then we have to supply the low and the close price furthermore we can supply the length for example 14 and yeah it will mess mess up with the data if we run it like that because we have to select the data for a given stock but yeah you see that it requires three columns so here we we going to define a function called compute ATR it will take stock data and here we will just calculate at which would be pandas ta. high would be stock data High the low then we have we need a close the length would be 14 all right and a little detail we are going to add of here is we are going to normalize the data while we calculate it so that would be ATR do substract first first we are going to the mean and then we are going to divide it by the standard deviation so ATR do standard deviation and yeah that will be our ATR indicator function now we can just say ATR create a new column called ATR and group by level one again so we are applying that for each stock however up here when we use the group by apply we need to add an additional argument to the group bio which is group key is equal to false because if we don't do that it will double the the date column so it will return another date colum and we have a triple multi-index with two date columns we don't want that so we just say group Cas equals to false and then apply this function and this will now calculate the ATR index normalized for each stock the next indicator we are going to calculate is the macd indicator and for the macd we're going to follow the same logic as as for the ATR indicator we're going to Define our own custom function to compute the macd so it will be called compute macd it will take the close price and up here we are going to say magd is equals to pandas ta. magd close equal to close length is 20 and then we would like to get the First Column which would be returned however in the end as well we are going to normalize the dat we're going to the meain the series and then we're going to divide by the standard deviation why do we do that right away we're normalizing the data because we are going to use it into a machine learning model we are going to Cluster the data we want to do that straight away and don't think about it later in the future so here we are going to do again group buy level one each scker on each stock group Keys equal to false and then apply compute Mark D this would calculate calate the mag the indicator for each stock all right we have an error why do we have this error that's quite strange oh okay I'm sorry I'm sorry I forgot to add the adjusted close column up here and this is driving the eror all right so we will return back I lock in the first corn that should be guys all right yeah now we have the macd as well calculated and normalized as you can see we have the data looks pretty good so far the only indicator we are not going to normalize is the RSI and there is a particular reason for that but you uh understand more about it when we come to the clustering part all the other indicators we are going to normalize and the final one is the dollar volume so we're going to create a new column dollar volume which would be equal to the adjusted close multiplied by the volume however we may want to actually divide that by 1 million for each stock because we know that millions of shares are traded each day and this would make sense as you can see now the data looks much better for the dollar volume right and that's pretty much it with uh calculating the first batch of features our technical indicators now we have a really beautiful data frame for each day we have all the 500 stocks we have the close price low open volume garment class volatility RSI Ballinger band ATR macd and the dollar volume for each stock and we are now ready to move to the next and the third step in the third step what we want to do is to aggregate on a monthly level the data and filter the top 150 most liquid stocks for each month why do we do that we do that to reduce training time for any potential machine learning model and experiment with features and strategies what is my idea here I would like to aggregate all the indicators so those five I would like to take the end value the end of the last value for the month as well the same for the adjusted close price and for the dollar volume I would like to get the average dollar volume for the whole month for each stock we can start actually what we can do first is uh take the data frame un stock the thicker level so we'll unstack thicker and then we're going to select the dollar volume column and if we run that we have the dollar volume for each day for each stock now right and what we can do is just resample to monthly and take the mean this should resample to monthly now as you can see we have monthly index end of each month and we have the average dollar volume for the month what we can do now is just stack it back into a MTI index like that and we can say two frame to make it uh a data frame with one column two frame dollar volume beautiful that would be the first step however for the indicators what we can do is we can follow the same logic but we need to select the exact columns actually we may create a list of columns so last call and this would be our list of columns for which we want to do the same operation however we would use the last method up here instead of mean and those columns would be C for C in DF do columns. unique and yeah the first element C for C in do in DF columns. unique if C is not in the following columns list so if the column is not dollar volume it is not volume it is not open it is not high low or close pretty much we want to do that only for the technical indicators columns so for those columns we don't want to use these columns for our aggregation we want just the Fe we are creating the features data frame in the end right we would use the dollar volume to filter out the most liquid stocks the dollar volume would not be featured in our model as well NE neither the volume or open low high close after we have defined the last columns uh we can actually proceed with the next aggregation we have done the dollar volume aggregation the next one would be for DF last C we are going to unstack actually we are going to un stack before that so we are going to unstack we're going to unstack and select those columns and then resample to monthly and then just use the last volume like that and again we're going to stack backwards into a multi-index voila and now what we can do is concut those two together we can say the following pandas cona and that would be axis axis one and boom we have the dollar volume the average dollar volume and the last value for adjusted close ATR and other technical indicators for each month now we have aggregated the data to monthly level for the features we would need actually what we can uh what we can add up here is a small drop and that looks much more beautiful and we can call that data and let's visualize what we have yeah that's our data all right and that's the first step with the aggregating to a monthly level The Next Step would be to calculate the fiveyear rolling average dollar volume for each stock and then you use this uh aggregated dollar volume to filter out only the top 150 most liquid stocks for each month how do we approach that first we can start by selecting the dollar volume like that so what we can do select the dollar volume and stack the thicker level and now we can use a rolling function with a window of 5 * 12 so 5 years and then we can calculate the mean as you can see now we have the rolling average mean for each stock rolling average 5year dollar volume for each stock and again we can just stack backwards and that's pretty much our dollar volume column what we can do now is we can assign it to the dollar volume column update to The fiveyear Rolling average for each stock awesome after we have that the next step is to calculate the dollar volume rank cross section for each month how do we do that we can say data Group by level zero or date so we can Group by on date for each month we're going to select the dollar volume and just rank ascending equal to false let's see what this would give us ascending and now as you can see we have all the stocks ranked by Dollar volume and the the ones the the guys who have the smallest dollar volume have the the highest rank so we want the top 150 and from here we can pretty much very easily select the stocks which are below 150 for each month and those would be the top50 most liquid stocks for each month after we have selected them we can just drop the two columns we can drop the dollar volume and the dollar volume rank because we are not going to need them anymore access equals to one and that's pretty much it for our we can assign that to data and that's pretty much it with our third step where now we have aggregated monthly data for all the features we would need plus the adjusted close price and we can move move on with the next step the fourth step and that would be calculating monthly returns for different time Horizons and add them as additional features to the ones we already have here okay so let's move to that step I'll just cut those sales in the middle all right why do we want to calculate the monthly returns for different time Horizons and uh add them to the feature set because we may want to capture time series dynamics that reflect for example the momentum patterns for each stock to do that we can just use the pandas data frame method uh percent underscore change and Supply the different logs my Approach would be to use logs for 1 month two months 3 months 6 months 9 months and 12 months that's uh like six different uh LS to really capture the momentum patterns how do we approach that let's first start by for example selecting the Apple stock I just want to make a little example so we'll select the Apple stock and let's see what we have here all right now we would like to calculate the returns for the following lcks for one month two months 3 months 6 months 9 months and 12 months right as well we may want to have an outlier cut off because we are dealing with a lot of stocks there will definitely be outlier values in the returns of those stocks what do we want to do we want to with them by clipping them what clipping does is that for all values which are above the outlier threshold they will just be assigned the threshold of that percent up the cut off value we may want to have it as a 0.05 which means the 99.5 percenti that would be our outlier cut off and now what we do is just for each log so for log in logs for each log we are going to create a column which would be the return for the given log for the given log month right that would be the column and then we grab the adjusted close then we just do percent change Lo so for each lck we'll calculate the following column which would calculate the given return and then we want to deal with the outliers right so that would be pipe Lambda so far we have the adjusted close and then we calculate the return for the given L and then we input that into the pipe Lambda X we can clip now we clip the return and we can clip the lower band the lower cut off we'll use x quantile and we supply for the lower cut off just outl cut off and for the upper one we Supply x quantile one minus outline cut off then we add one to the power of 1 / by the log and we substract one in the end and that should be it guys now we can see that for our Apple stock which is G right we have the one month return two months return 3 months 6 months 9 months 12 months Etc how we can extend that we just use the same approach we use for the at and the macd indicators we will create our own custom function and then we you just use the group by apply methodology for the bound data frame so calculate returns it takes DF for example and just move that a little bit return DF in the end however up here we have to change that to DF this one to DF as well and that's pretty much it our function and now we can say the following data equals to data. group by level one because we Group by on the thicker level so we can say uh Group by ticker or level one let's say level one and then we want Group keys to be false so we don't have two uh date indexes assigned to the new date frame so Group keys false then apply we apply the calculator T function and in the end we may want to drop the na values and yeah this would take some time and yeah that's pretty much it I mean now we have added the return features as well which we would use to capture momentum patterns for each stock and that's pretty much it with the fourth step the calculating the monthly returns for different on Horizons and we can move to the next step which is really interesting actually adding even more features to our data set that would be to download the F French factors and we are going to calculate the rolling Factor better for each stock in our data all right let's move to it okay so in this step we're going to download the farm French factors data and calculate the rolling Factor betas for each stock in our current data set so we want to introduce the F French data to estimate the exposure of our assets to commonly known risk factors and we're going to do that using a regression rolling OS model the five round French factors namely Market risk size value profitability and invest M have been shown to empirically explain asset returns in the past and are commonly used in the asset management industry to assess the risk return profile of different portfolios so it kind of makes sense to include them in our current feature data set how we can do that we can use the pandas data reader package we which we imported as web we can use this package to download the F French Factor models but before that we may want to to take a look at the we might want to take a look at the data so we can Google it Farm French factors and up here you can find the canid French data Library just click on it and this is the part we are interested in in FAL French 5 research factors 2 * three so those five factors that's the data we are interested in if you scroll down a little bit you can find the daily data right here you can download it as txt or CSV file as well you can check the details they have monthly returns and annual returns we are interested in the monthly returns as our data is already on a monthly level right so what we can do is uh we can say the following web. data reader data reader and up here we have to supply the name of the exact Factor so the exact file so I have prepared it already that's the name then we say F French then the start date we want 2010 this is returning a dictionary with two keys the first one is the monthly factors and the second one is the yearly factors that's awesome so we want the monthly factors only and that's the data we have 164 uh months up to August 2023 that's pretty good however I think we don't even though the risk-free return is pretty solid right now it is out of the scope of this tutorial so we'll just drop it we say drop RF AIS equal to one and and that's perfect we can call this uh assign it to factor data and yeah that's our Factor data let's check the index the index is uh monthly yeah all right so I think we have to fix the index as well so we can call of pandas to daytime and Supply the index let's see if this would work okay we have to use to time stamp all right okay so now we have fixed the index as well as you can see now it's it has the year the month and the beginning of month date however our data is end of month that's one thing we have to fix another one is that I see that the factors are in percentages so we would have to divide them by 100 how we can fix that we can just say resum P to monthly and get the last value which would fix immediately the issue yeah with the beginning of mandate next we can say divide by 100 and that's perfect we just assign that to factor data comment it out yeah next we want to fix the name of the index to be just date and the next step would be to join with the pretty much join with the one month return why would we want to do that because at the beginning of each month we have the factors and then we have the return of each stock at the end of the same month so now we have fixed the date of the factors we can just join with the end of month return and then we can can regress them and take the beta right if the factor is predictive we have it at the beginning of the month and we will regress it with the return of the end of the month so we'll get the the the beta how we can do that factor data join and then from our data we can select the one return column let's see what this would give is that's perfect now we can sort the index and just assign to factor data what we may want to do here is to double check our work we can select two stocks for example apple and let's say Microsoft and we can double check the return yeah the return is different but the factors stay the same looks like we' worked correctly and we are ready to move to the next step in this step we are going to filter out stocks that have less than 10 month data why are we doing that because we are going to use rolling window for the regression of around 2 years 24 months and stocks that don't have enough data would actually break our function so we have to remove them from the data set how we can do that we can say Factor data. groupby level one and then just call the size method and now we can see how many months of data we have for each stock okay guys and I just realized that we have made a small mistake somewhere because we have only 23 months of data for each stock and I had to go back through the code and actually f figure out that on this step uh this part was missing which was the data dolog selecting all the rows and then the dollar volume column you can just rerun this sale and you can see that here our our first month is in 205 November and after that it was 2020 so when you add the dot loog and selecting the all the rows and the dollar volume column you get the fixed data then we can rerun as you can see up here it's 2021 the first date if we run it it should be much backwards in the past yeah so 2017 31st of October and we have to do that again for the factor data two and if we run now yeah now we have 71 months right and the idea here is uh that we remove all the stocks that have less than 10 months of data how do we do that we can just assign that to observations and then we can save valid stocks that would be observations that would be all the stocks that have more than 10 months of data observations yeah those are our valid stocks and now we can just use that as a filter so Factor data would be Factor data again Factor data. index do get level values we get the thicker values maybe I can show you what this is returning this is returning uh yeah an object with all the the stocks we have in the thicker index part of our M index and then we can just say is in the valid stock stocks and this would filter out pretty much everything we have to add this part okay okay so I think oh yeah I missed this part should be all stock index those are the stocks we are going to remove so if we take that out yes so now as you can see we hit before we hit 10,21 rows now we have 10,250 we've removed around 51 rows and yeah with this step we are now ready to calculate the rolling Factor betas we would like to do that simultaneously for all the stocks in our Factor data we can just use the same methodology with Group by and apply a function that would be Factor data. Group by and then we're grouping by level one by ticker Group keys should be false and we can put that into brackets so we can continue on the next row so apply Lambda X and now we want to use the rolling regression actually we may want to explore the rolling regression rolling ORS python it takes the endog and exog all right we here Supply endog and exog so the endog would be our return column right so X return one month and our xog would be everything else what we can do here is say x do drop the return one month column and this would return all the other this would give all the other columns without the return and actually we can add a constant here on the spot so we add a constant and I think the next one we have to supply the window we decided the window to be 2 years right so that would be 24 right 24 months and there is another mean observation the minimum number of observation required to estimate the model we have to supply this one as well this one is a little bit more tricky we have to supply here to have at least the total number of columns plus one so that would be Lan x. columns plus one and this should be our model now we have to say fit and then params and then we can drop the we can drop the constant because it will return constant as we added a constant up here right and I think this should pretty much work except that maybe sometimes we would not have exactly 24 months of observations but we still may want to run the regression so what we can do is use as a window the value which is smaller than two Val so it either 24 months or we can use the number of rows we have in the for the given stock right and we know that we have stocks with more than 10 months of data so if one of the stocks have like 15 months of data you just use the 15 months as a window instead of 24 and yeah I think that should pretty much be it right okay series object doesn't have fit why right maybe okay so that's pretty much that's pretty much uh our rolling Factor betas we have calculated them now we can assign them to betas and that's our rolling Factor BS guys in The Next Step what we want to do is to join them to our current features and with that we have our Full Features data set but before we join them we have to think about a little bit now we have the rolling Factor betas where we used the factor at the beginning of the month and the return at the end of the month so this beta we would actually know at the next month right at the end of the month we go we'll be able to run the regressions and have the betas but we'll have them in the next month so we cannot just blindly join them to the features data we have so far what we have to do is to shift them with one month forward before we join them to the data because these values we would have no not known in the same month we would know uh we would know the mon the the rolling Factor better for example for the end of October we would know them in November what we have to do is we have to shift with one month forward on the ticker level so for each ticker not like the whole data frame if we just say betas do shift it will run it will shift with one row downwards and for example the value of Verizon will come up up here right just see that so Verizon is 0.3 if we apply shift it's now here so that's obviously not correct so what we have to do is we have to first group by and we can do that group by thicker and then we can shift that would now uh do it correctly we can grab that and Supply it here and data will be equal to this and the next step that we want is to impute the missing values of each factor with the average for that factor beta how we we can do that we can say first we can create a a list with the factor columns factors would be just those five columns we can say data. log all the rows factors we'll just select all the factors and all the rows and then we can say the following data. Group by ticker for each ticker we'll select the factors again and then we'll just apply a Lambda function Lambda X where we will F all the missing values with the mean of this Factor but I think because we are doing grp Group by apply we have to add the group keys to be false and now we should have fixed our missing vales issue voila so now we don't have any missing Valu so all all the nas are imputed by the average for this factor and yeah now we can just say drop a if there are any and we can say data. info to see our final result and that is now beautiful guys this is our features data set however I see a column we don't need here which is the adjusted close we just have to drop it data is equal to data drop adjusted close axis equal to one and and this is our features data set guys so we have 18 features at this moment we are now ready to apply machine learning models from here on what we have to decide usually is for each new month we have to form a portfolio with some of the stocks from our uh data set so as we know we have for each month we have the top 150 most liquid stocks and now now at the end of each month we have to decide the stocks we want to have in our portfolio for the next month that's where we can use a machine learning model first we can use a machine learning model to predict which stocks to include in the portfolio as well if we have a long short portfolio we have we can predict which stocks to be long and which stocks to be short but in this course we are just focusing on Long Port foros so we can use a machine learning model to predict which stocks as well we can use machine learning model to predict the magnitude of the position in each stock so what is the weight in the portfolio and the other way is to use a machine learning model in our case a nonsupervised model to decide which stocks to use in the portfolio based on grouping that's why we are going to use a clustering algorithm a KES clustering to keep things simple because from this point on things can get really complicated but yeah that's so far was the preparation to get the data to fit into a machine learning model guys and yeah I'm really excite