How do Algorithms Generate Recommendations? | DataHour by Varun Behl

Key Takeaways

girls are getting so one such example could be that which movie you would like to watch and then how this uh data gets processed is there are first the top 10 recommendation that is how people are watching and what are the top searches people doing what are the top movies people are watching on these algorithms but then based on your likes and dislikes of genres which movies you've checked just a small snippet of a movie and all these data points get captured and they basically tells you what is the best fit next movie for you now the kind of data that these algorithm use are classified into two sections one is explicit data and the other is implicit data now explicit data is that is basically captured based on a input that you are giving to the app let's say you watch the movie and you rated that movie or let's say you were on Amazon you bought a product and then you rated you commented how it was you reviewed that product that is an explicit input that you are giving to that application whereas implicit is something that they infer from your behavior which is something like whether you clicked on a particular product you scroll down that particular page or the time spent of that page what did you search on that particular search bar of that app what kind of genres you like so all these are inferred from your behavior so everything that is inferred and you might not even know that that particular thing is tracked it's basically track so whenever you go to an app or a website there's always a pop-up of whether you want to accept all cookies so these cookies basically track all your behaviors even if you're scrolling out the screen which part of the screen you are clicking which adds your clicking all these kind of help in categorizing you and what kind of you user you are there's this basically all the set of implicit data and explicit data is basically what input you are giving so both the datas have their uh importance while building these kind of algorithm implicit data although has a very large importance in building a very sophisticated algorithm because that is based more based on your behavior and even you might not be very sure about what your behavior is or the on such apps but when you run these data through a particular algorithm it can basically classify your behavior and map it to similar pattern with other users on the platform and that's how they recommend you a similar thing that a particular other user would have uh like selected also now if I have to divide collaborative filter it is based on two approaches what is memory based approach the other is model based approach I will be further deep diving what these approaches are uh like as the memory base has been classified as user base which has probably two techniques either correlation or cosine similarity and even item base that is also caters to based on the similar technique of correlation and cosines the model based approaches uh capture clustering which is also called sometimes as content based collaborative filtering and the other is Matrix factorization and Matrix vectorization is one of the most used uh algorithm across every company or any platform that you would see they have just enhanced it to their use cases more and more and that the more and more data they keep on inputting to this uh kind of approach the better the model gets and the more better recommendation you see another additional to Matrix factorization now is the Dural part of it so the neural Nets where they are used for Matrix fertilization that even enhances your experience of these applications now what is user based uh collaborative filtering or memory based collaborative filtering aspect where a user is evolved so when your preferences are matched to another user or what he is watching so let's say you watched uh you you ran three songs on uh YouTube and there is another user of the app who has watched those three stocks plus two three more songs Baby of a similar genre baby of a different genre so what recommendation you will get that because your pattern matches to this particular user you will also be recommending those songs that you haven't seed but that user has already seen it also goes down to movies or any trust products that you scroll on Amazon or Flipkart any app that basically gives you a suggestions of anything that is how it works now item based collaborative filtering is when let's say you watched or read a particular book or you bought a particular book called Amazon now the Jordan of that book or the category of that book also covers a lot of other books so when your item your matched item wise or your mapping is the item wise then you are recommended based on item and then let's say if a person have given a very short example of if you've seen that a lot of the Rings book then your recommendation will also be something related to fixtures that could be a Game of Thrones book or a Harry Potter Facebook so that's how it's it works and when these two techniques let's say you you get an output from a user based approach and you get an output through an item based approach they both are collaborated together they both are weighted that which uh for which particular item or which particular thing you are getting a higher uh score that is how your recommendation is sorted into so that's how these kind of approach works it's a very basic concept even if someone has to recommend something to any other person we also give a word of mouth suggestion based on these kind of things only that I like this thing and you have a choice similar to me maybe you should also try this because I also like this particular movie so that's how collab like user based approach work moving towards the clustering or the contact base approved though for this approach as you could see in this particular thing the data or the user base or writer based decrease the data I was using was only a SQL input uh basically it was either your rating for a particular movie have you watched that movie or not watched that movie Have you listened to that particular song or not that listen to that song so basically if you want to think of your input it's like one or zero it's like a binary input that you are feeding into this user base or item based approach whereas for a clustering or a Content based approach your data can be multi-fold it can have ratings it can have views it can have your page clicks timespet dot Pages uh reviews by other users and cupboards that you give for uh so it basically covers both aspects of explicit and implicit data and when you do a clustering approach it basically categorizes similar or classifies similar users together so when you have uh when you run a clustered model and you get a say cluster of let's say type users who watched it's a six movies say but there are few more movies that one of the user hasn't seen so it's an easy recommendation that if he has liked those movies there is a very high probability that he's gonna like those movies as well uh so yeah this uh I was talking about plastic now moving to the Matrix factorization all the the challenges of the above three approaches are that if your data is very sparse or let's say very less inputs are in your data the above three approaches tend to fail like if you don't have a rich data like if you have uh too many dulls in your data let's say the you are a very new user on the platform and there are you only hardly watch couple of movies on Netflix so how should that couple of movies help you suggest a new movie out of it so what kind of data is needed for this and how does this approach differ from other approaches so the kind of data that goes in Matrix factorization is even larger than it is in clustered or uh item your user based approach it basically has all the data inputs that you've given above three Techniques plus that what was your last watch movie Last was solved lost heard solve where did you scroll how are you scrolling how much time you were uh like it's the every aspect of your behavior is the sum or the other form is captured in the data and all these data are basically either created into a binary format or a continuous variable format because categorical for input or categorical column does not work for this approach so basically you have to create few more Columns of uh to feed into this particular approach but your categorical data will not work with this particular approach now the basic aspect that it tackles is the sparse is sparsity in the data and even scaling it to a large audience so or for all the above three approach it takes a very long time if you are running let's say on a billion user base so YouTube or Netflix or any other app Amazon they have a huge user base of their data and there are a lot of prospects that keep on coming so you have to even suggest for a person who's coming for the platform for the first time so what kind of data they use they use like let's say they had suggested for a hundred new users coming to a platform they had suggested X items and what was their response for those X items they use that to trade on the new prospect that is coming wherever whereas when you have to do this or the above three approach that is not possible because it primarily caters to what you've seen and what you have selected it's more on the explicit side of data not on the implicit side of the data it's not relied on how other users are doing foreign moving forward I will first like talk about user base and item based approach although I will be combining them and talking about cosine similarity nutshell and how this particular approach helps in creating a matrix that is further using recommendation right now cosine similarity approach uh uh you guys can also drop your questions and I will try to answer all of the questions uh once I cover this particular slide and even at the last of the session we are done approach basically helps us in giving a particular score of how two users are similar or how two items are similar so the data input that is required is basically will do a DOT product Matrix of this so if you know about what is a cosine function it is the thing it tells what is the angle between two vectors so if the angle between two vectors is zero that is they are highly similar you get of course a value of 1 as an output whereas if they are highly differentiable the output will be the negative side it could be minus one so that means you are highly different from that particular user so what I have kind of data that you can prepare there are multiple ways to prepare data for this you can use any kind of metric that you feel is a good metric to compare to users so let's say if I am giving you an example of a project that I had worked out was how which uh Telecom plan to recommend to a new user coming or let's say someone was on a very old plan of a particular Telecom company if they have launched a new plan whether it is a fit for a river or not so we match the attributes of his usage Behavior how much data packs he use and everything that kind of data helps in understanding whether a new plan what kind of new plan can be suggested to this particular uh new user so a user item Matrix is created basically the thing you have all the user one user to user three data on your as your rows and the items is basically if I'm talking about Netflix it will be movies if I'm talking about Amazon it could be products if I'm talking about YouTube it could be songs so the value in these would be whether you've watched it or not whether you've rated it or not or it could be even what is the rating you have given so but it has to be in all the columns it has to be consistent for this approach to work it cannot be you have have a binary in one column for one particular movie whereas you have ratings for the other movie in a different so this this particular thing might not work because you have to be consistent in terms of having a binary data for this now what it does is it when you apply cosine similarity for from two Matrix let's say what was a you you use that Matrix which have watched very few uh listen to very few songs on YouTube and whereas other users is a good Rich data where these have so see uh or watched a lot of movies or see lot of videos or heard a lot of songs on YouTube so this particular aspect of the one of the data has to be very rich and the other data can be partially populated where your recommended you want to recommend these users now it will when you run a cosine similarity or this particular thing it's basically nothing but just doing a matrix multiplication and capturing whether how much you are similar to the other user and it will create another Matrix which is uh which will give you a score between zero to work but then how much you are similar to that particular user so let's say if you can put any threshold in there I have let's say I've taken an example of I put a if a cosaic score of 0.6 is there you get so cosine similarity works in the similar way how your correlation would work yes a correlation would work so you can even use correlation how you run correlation on your Ed and all it's a similar approach it's just that you need a score of similarity with another user so you can use cosine or either Pearson correlation so if I talk about cosine similarity you can replace that with peers of correlation also both the approaches will kind of give you a similar result now um if if I talk about the second Matrix that you'll get out of it is basically your score metrics I've highlighted the fields where the they you match to a particular user uh uh it shows you a particular like let's say if I'm talking about user phone it you manage to see one user because you got the score greater the 0.6 also I will see 0.6 was given to uh also against C6 so what are these C6 users or C1 user watching so let's say I've picked an example of C6 user he was kind of watching P1 and P6 products uh and then you basically suggest they have a score of these are the kind of project that this user was watching that can be recommended to you so if you go back to what I was talking to this is more of a user based approach of collaborative filtering because here you are looking at what a user was watching and that's what you can suggest to this particular user also whereas it has also matched to C3 also but C3 maybe had a plan different uh sorry baby had watched a different movie or something else you are fairly magic within so your recommendation can be audited that way also so when you run this exercise on a huge set of users you can basically get a matrix out where you can sort them based on the cosine score that you are getting and when you score them it would look something like this so let's say for a particular product you were giving R1 basically R would be that was recommended as the first thing R to be the second thing at R3 big the third thing that was recommended to you now if you scale this on a large data set it looks a much more enhanced and much more uh understandable format this is for lecture where you take an example of six user but these kind of algorithm dot a very hard large set of data where you have a million of data or which you look which have already been out the platform and a thousand set of users for which you want to test this particular thing so current plan users and new plan users I've just taken an example of let's say it was a movie data so someone who has a new plan Uh current had already a new plugins for some you have to recommend a plan so this is the approach a user to user Matrix would look like this will give you what a new subscriber they would pick what kind of plan they would put what kind of movie they can click so you can interchangeably use plan against any kind of item that is there so it it it will it will correspond to if you are on the app that is an e-compass the plan will correspond to a product if you are on a hotstar kind of platform the plan will call to your subscription plan or something like different options that are available this is I've just suggested the formula that goes behind the cosine score and that's how your recommendation is generated uh this is like a basic Niche formula that you will find across every uh cosine similarity recommendation approach that has been used this this is the basic uh approach that even if the you can replace it with the peers and correlation score so that will also give you a similar output for this particular thing it is the third rule is that you should only suggest top three recommendation out of this because there are a lot of user which you match with but you have to size it down to what are the top three recommendations so that depends on how you sort the ranking when you get the score out of the cosine similarity uh Matrix again last pointer I have suggested that you can even relate or replace this with the Pearson correlation approach as well it I'll just probably check uh if there are many questions all right that is of right so moving forward I wanted to touch base on a very important concept because this will help you with understanding how Matrix factorization works and I had touch based on this kind of approach in my last session of PC as well on how these related factors or principal components are created or factors are created out of a large set of columns because if you don't concise your Columns of different values different kind of attributes different kind of data types into a simpler form of a continuous variable Matrix your model is tend to fail more so you always it's a good practice to concise your data as small as possible and PCA later Factor Matrix factorization SVD single value decomposition these are few approaches that are widely used to reduce the complexity of your data reduce the size of your data but the information does not lose so the information retains and but you are able to run your model faster but it is in a very concise form the later factor is basically a factor that is created out of a large set of columns but it indicates to a particular attribute of those columns that have been selected so I've taken a very basic example of let's say this is a subscriber or a journal attributes or like a if you if you go on Netflix this is kind of a data that you might see like these are genres your clicks review types aspect of that particular movie or or the web page and number of tags you've watched this is a basic uh columns that I've thought of it would be there in the data that is been used for a movie recaptation kind of algorithm and I got and these are it's this is a this is just I'm taking an example of one movie so if there are let's say 10 columns for one movie there could be a thousand or ten thousand movies on a platform with 10 000 movies would have all these Tech columns so you can imagine ten thousand and tenfolds how many columns would be there and when you have such a rich data it is always advisable to go with a matrix factorization approach because you can you always have a capacity to reduce the complexity of your data and all these are not available now how these related factors help is uh all these let's say actual history or Republic I can already see these catered to journal and when I do run a latent factor analysis or a PCA or this particular data it will give you a smaller data set of uh let's say what color which I can let's when I checked values or whether it's a which weightage this particular compilator which for this particular factor is giving to it will be related to these four columns because it's tagging it well to all genres because the how user have been tapped to this they show a similar pattern in it all the columns will indicate that this is probably related to a journal column although when you run Matrix factorization this part of the entire algorithm is a black box but this is what is running if the background type taking a very rich example to help you understand the concept of it but that's how your latent factors are created it's basically a combination of columns that help you Decay to a particular attribute and you just concise them now engagement could be related to clicks and review that you are doing interaction could be related to the time you've spent with the page or the watching the movie or the number of times you've watched a particular thing so that's how you drill down from eight columns to three columns and probably an 80 000 column can even be further down to three or four columns so this entire exercise helps you reduce the type that you would expect the driving your model and get the output out so it's basically a utmost uh necessary thing to be carried out before you're running this kind of algorithm uh now I'm taking an example of ALS Matrix factorization of it stands for alternating least Square Matrix factorization approach it is based around the concept of how your PCA mathematical approach is based on it's based around the single value decomposition principle which I will obviously be walking you through when I'm talking about this approach but it basically what it does is it takes a large Matrix creates it predicts the value where it is there are null values and helps you create a new Matrix with the values which were not initially present in the data the other approach when you're talking about PCA and how it is related to the PC approach it is basically it also creates the latent factor or the components that is also part of that particular approach so that is where uh these two approach have some uh overlap in between them now Alf is an iterative optimization process well I am saying iterative is basically the Matrix keep on multiplying with itself till it till it is able to retain the Minima so if you if you know about the concept of uh array regression model or any least Square model basic regression decrease they always try to attain available is basically when your error is variable so even this approach tries to generate values for the user that have not viewed a particular item or not seen a particular item or not watched a particular movie and give a particular value to those corresponding users and how it does it keeps on turning into an iterative look keeps on generating a best fit value for those users and whenever the error for those user tends to be delivered that is where the algorithm stops and it gives you an output so moving forward uh subscriber feature Matrix have again taken an example of let's say you are on a Netflix app you have a number of subscribers you can divide you can call that your data is divided between trade and validation but how I have divided it is basically validation set is somewhere where you will be suggesting those movies a train is where people have already seen such set of movies and these movies could be new movies well I'm talking about the data set how it would look like it will have some features like as I was talking what was your last viewed movie what was your movie that you clicked on what kind of genre you watched reviews ratings types paint and every other aspect of your behavior on that app can be basically covered with other features now item Flags is better this movie should be suggested or not so for the trade subscriber it would be liked by those users at the like you can let's say if there is no like people are not liking that movie explicitly you see whether they've watched it fully or not so that is also sometimes as a taken as a proxy that that person has seen that entire movie is basically like he sticks to that movie because he liked so that's also considered as one flag um when you are kind of building these algorithms you can create always take these kind of proxies when you're building these the latent factors as I again told you there were genre available they were engagement metrics available and interaction Matrix available so those latent factors help this Matrix factorization approach understand I am an X User to which other user my pattern is badging my user behavior is matching too so that will help them understand that if that particular the user has seen these set of rubies they should also be recommended to be in some corner so now I've given this example the later factors are also tagged against the item Flags is basically how this Matrix is Multiplied and keeps on regenerating the values now I've given an example of that the null values over these basically there were no values whether you watched or not it gives a probabilistic score between 0 to 1. this approach will always give you a value between 0 to 1. only if you have a continuous data set again if you don't take categorical value and probably throw you an error when you are running it on python or SAR but if you are giving a continuous input between 0 to 1 or even putting a binary data set that is the best possible scenario and then it will also give you probabilistic output between 0 to 1 or what is your probabilistic score for that rule so for the user that were not recommended anything or had not seen certain items they were recommended given a recommendation score for a particular movie and then it will pop up on their screen that's how the suggestions that you see on YouTube of songs or videos everything it goes through this particular kind of algorithmic approach and that's how your suggestions are ragged so whatever is right is given a very high score to you is Right dot top right subsequently the lower kick options cover your screen uh now as I've told you the subside it it is you can basically understand the subscriber feature Matrix is created on a huge data set uh there are other uh I've also taken an example of um a Telco industry as I've given you the previous slide that they could be based on the usage Revenue device type what you are using so maybe people with a similar device May tend to kind of they watch YouTube movies of a higher quality of something like that so that is also one aspect that is considered in these kind of data so someone who has an iPad compared to an Android mobile their recommendation of what kind of movies they are you videos they are recommended also differ so something's done better on an iOS platform some things are different on an Android platform and in what quality they are available also matters the app experience so obviously or two platform also differ so that also uh gives you a certain kind of recommendation that device and the platform OS platform that you are using also governs uh what what recommendation you are going to get uh the last thing that I want to tell you about this approach is that it runs come very well or the very sparse data so I did not have any recommendation or interaction of these user that I had to suggest recommendations for they had not interacted with the platform but you have to recommend something to them so that they gauge with the platform so basically recommendation algorithm are focused on the prospects more because that is from where you want your engagement mode some of the most of the users on the platforms are regular they keep on coming back so for them the recommendation algorithm works pretty well they tend to take more recommendation suggestions and even those recommendation might not be the best fit for them they tend to be more engaged with the app because they have already fallen for the uh the UI or the experience that is there but those who don't retain well for them the recommendation algorithm uh has to be powerful foreign I've taken an example of how an existing system in let's say YouTube would be working right now so there is obviously a video Corpus that is the entire video set that is there there is a user history and context data how whether the user has been engaging by cupcake or videos and all those kind of data and this then there is something called candidate generation generation is nothing but what Matrix I was creating in this approach as the last Matrix so if let's say a particular user has watched something there is a huge video Corpus that he hasn't washed then what would he tend to watch what are the scores that you so candidate generation is something that is being taken care of that if you are watching a set of items another user are watching a set of items what how your candidate generation Matrix would look like so that is this is nothing but a data set only which is created based on uh existing video set that you haven't seen and your user history kind of approach of this you are ranked because there are other candidate sources which are basically the other users of the platform and they have already seen the other video features data is basically what kind of videos you're watching in what aspect ratio you like to watch all these factors they might not be very highly dependent on your user Behavior but these kind of things slightly impact in improving your recommendation so as much data as you can give it could be related to video features could be related to what kind of Journal that you are watching all these at your behavior last watch video lasted so all these kind of collaborate together and generate a rank for a video that's how you get a list of suggestions on them and talking about the advancements that are being made are in terms of the neural side of it so as I told you when I was talking in the previous Slide the latest factors were only created for the items or basically your behavior of the website it was not later factors what got created on the users so how users later factors can be created as basically a user that are interacting from the similar platform uh watching on the TV screen some are watching on a similar device so you can consider a very basic example that iOS users are in one later Factor Android in one device different device iPad and desktop others so that's how uh basic related Factor would look like whereas other aspects of how your how much time you spent on the website that can also help you relating the user what geography you are part of and when you enroll on a particular platform they also take a lot of details they get your email ID they get to know whether you are a working professional or not because they also tap your data from different sources as well so basically that you a user related vector or a user's Corpus of similar user can also be created now when you have a user Corpus of uh which you can create later factors out at similarly diet and factors that can be embedded together and that approach can only be done not by an ALS which is basically you need to have a user writer batteries whereas in which you can have a both item Matrix a user a user related Vector Matrix and an item latent waiter vectors and this can be combined in a neural net forward and those layers can obviously like deduce you whether this particular movie or this particular item is best fit for you or not so it's a very complex approach we can probably have another session on talking about how this entire architecture looks like what is the mathematics involved behind this dual collaborative approach but for uh letting you know there are advancement paid on this side and as in more your data increases on this platform these neural model keeps on get enhancing their efficiency and you keeps on seeing the recommendation that you are getting on Instagram or YouTube or Netflix are improving day by day so that is how uh the board I respect on the website or applications the more better recommendations your keyboard gonna get uh moving forward um so yeah this was more about more or less about the recommendation system and algorithms um the two approaches you can there are a couple of data sets available on kaggle or which you can primarily apply these techniques and see the result for yourself how these work and uh cosine similarity is the first thing you can start off with to basically understand uh how this thing work and moving forward to ALS metric factorization and then to neural uh approach so all these have uh different mathematics behind involved and you will get to know once you run these kind of algorithms uh by yourselves sure I'll probably take uh take some questions now um so again latent factors um when I when I talk about latent factors basically if you want to concise a group of columns into one column basically I have a lot of columns that cater to genre of a book so there will be like 20 dollars of a monkey but that all can be combined into what particular column so concising them from 1 50 to 1 is how you create a latent factor at that later factor helps in governing what you are recommending and it also with your reducing the size of your data the speed of your model increases so that is the purpose behind this how to solve a cold start problem please give some approaches so yes cosine similarities what go to approach you can have prepare a data in which you have users or in all the rows and your items or let's say movies on the columns let's say you there are 100 movies uh 100 columns there and just the value of one or zero in them and you will get to see the user who have similar taste or if movies are also recommended a further movie that is also of their time so that a particular approach I am not sure uh where you would know about Association rule where they go to so probably I'll look up into this but it's kind of not um uh related to the concept I was talking about here in the session yeah um if there are no questions I think we can end the session and I would like thank you thank all of you guys for uh um being part of this data R series and being part of the multiple sessions I have been conducting with analytics with you and I hope to see you soon for the further upcoming sessions that will be also along the same line of machine learning approaches and the use cases that are being solved across Industries thank you so much thank you Varun and uh for everyone others uh there's a feedback poll uh in which you can give your response and uh the next data all session is on 21st of January uh kindly to check in the analytics with their website for more such data our sessions I hope you guys have filled in the feedback Bowl if not I request you to please fill in the poll about the feedback as it helps us to conduct more suggestions and if you wish to conduct a webinar or uh are facing any difficulty in registering connect with us at enter at the rate Analytics vidya.com and the recording of this session will be available with two days within two days on our YouTube channel now that is analytics video and we'll be back with another session of data award on 21st December at 21st upper January and the link is in the chat section till then bye bye and keep learning