🔴 LIVE 🔴 MATRIX FACTORIZATION TECHNIQUES FOR RECOMMENDER SYSTEMS

Key Takeaways

all right let's see if this uh this works was a while since I did uh a live stream so let's see all right seems to be working yeah I had some issues with like delay and stuff so hopefully it's good how is everyone doing is there anyone here um all right yeah I mean the the plan right now is just to uh I'm like doing some recommender systems uh and okay basically I just wanted to read uh an old paper actually on like the first one on Matrix factorization so just wanting to take a look at that and figured you know why not do it on on stream my plan is actually to I've done a couple of recommender system videos but I want to continue doing it so so that we uh hopefully like create like a a playlist for that in the future but you know now I'm just learning I don't know where the chat disappeared okay all right so this paper is really old it's from 2009 I think but hey if you like look at the citations this like is one of the most influential papers and you know Matrix factorization used all the time uh in industry for recommender system it's like the go-to although it seems like you know deep learning approaches uh with neuro collaborative filtering and just that kind of stuff has overtaken but as a quick fix it seems like or like a a quick solution it seems like it definitely works well so I wanted to uh take a look at that and uh you know didn't all right so the next Netflix prize I didn't think that was actually in 2009 I thought or I thought that was uh later I wonder what year that was into oh it was in 2009. well that's actually a really long time ago good morning guys I'm doing well just reading some papers and chilling having a cup of coffee I wonder what they did before this so they did nearest classic nearest neighbor recommendations curious how that looked like probably opened this in the so usually I mean it's a good idea just to skim the paper first um I don't know what they are like mostly focused on here I'm curious just like learning of their like how they use the right Matrix factorization and how they yeah what type of Matrix factorization they use because there are multiple different ones all right so they talk about collaborative filtering content filtering so they go through the basics I guess Matrix factorization methods so to my knowledge it seems like implicit feedback has become more important than expl I mean it it is just because you have such a vast amount of it but that systems now rely more on the implicit feedback than the explicit and that yeah you can basically remove the like the explicit but I guess you know both are used yeah singular value decomposition thank you so I guess like what I mean what you need here for SVD you need the user item rating Matrix and that's basically all you need but I'm imagining like how big that is like it doesn't scale right you can you can't major you can't do factorization on a huge Matrix so I'm thinking maybe there's a way of doing the factorization that is with a gradient descent methods uh that's SVD I'm actually not completely sure how it works but uh if you're using you know like linear algebra directly to factorize it then it seems like it could be a problem just because it's so large I don't know maybe we can um now let's ask we can chat with with uh GPT uh yeah I'm doing uh um that's what I'm doing for work now uh recommender system so just makes sense to to you know learn about it torch SVD low rank I have not looked into that is it from this paper maybe uh the paper we're reading out no huge sparse matrices all right I mean how huge can it do any oh that's interesting I'm curious like how it works in practice um uh PhD machine learning um most commonly used in Industry we can just get an idea so I mean I was expecting SVD to be the most popular uh what are the pros and cons how and how does it deal with huge matrices I think for SVD right you have the constraint right that each like for the two matrices one of them should have every row to be orthogonal and then the other one has every column or it might be vice versa but that that is a a requirement that you have otherwise you can't do the interpretation I guess yeah so this is what I'm worried about uh like how large four large matrices truncated SVD randomized tens of thousands all right let's look back at it so so this is what we are optimizing for the rating uh the rating we have for that user and that item subtracted with the those two matrices multiplied and then I guess there is our regularization term okay so they do use um yeah stochastic gradient descent to learn this but also all alternating least squares like I'm not exactly sure what the limitation is or why I mean one thing that's difficult with Matrix factorization right is that you're only based on the actual Matrix with ratings it seems like if you're using like a neural collaborative filtering approach then it's very easy to just mix and match like new inputs uh you might have a hybrid system where you're using content based system where you can just put in user info and so on but here with Matrix factors in you're only based on the uh like the actual rating Matrix so there might be some some problems there or like some difficulty expanding it thousands are already quite large I mean so then it like it doesn't scale right if you have uh at like thousands uh I mean you you if you're building a system it seems like you have to build something that is preparing to have thousands of users uh if not then it's not a good system I would say so but I know that it's used a lot so I I don't know there must be ways to circumvent that I feel like by the way let me know like if the uh I was I had some audio sync issues uh so let me know if uh if it works like if there's a delay or something thanks uh uh Ashish I appreciate it thank you just want to get like a rough idea of this paper first just skim it and see you know what are they focusing on um all right it was in 2006 announced uh contest so like the I think the winner of this was a matrix factorization method for the Netflix prize but I wonder how they put it into production if it's actually that limited because Netflix has way more than thousands of users so I'm really curious to read about that Our Winning entries oh okay so these were uh these were the people that won our team entry took over this the spot in the top competition and one eight point four percent better than Netflix okay all right so I mean they uh it works right okay I'm curious if we can find that uh Netflix prize competition GitHub foreign factorization user item Matrix number of latent Dimensions I guess I parameter right learning rate so we have p and Q we have biases and create a list of training samples okay so it's a list of just the uh the uh user and item values and the rating and then perform stochastic gradient descent for a number of iterations uh we Shuffle the samples cell.sgd [Music] so that is self.sgd okay I have no idea self.sgd that seems like oh there it is I mean that's a weird way of um of okay so we just initialize it and then we train it for a couple number of iterations um and then MSC SGD it seems like um I mean here you've done like the back prop by yourself which is I mean cool but you could have this is in numpy I guess so I guess we could do that in torch and it would be probably a lot easier but it doesn't I mean it doesn't look that difficult uh like the code I guess that is maybe one good thing it's a it's a really old paper by the way but this is the I just took the first one Matrix vectorization techniques for a commuter system and just took the first one all right so let's go through a little bit more detail I guess you know what we want to learn here is um just get an understanding of that Matrix factorization which we you know we already have gotten an idea but a little bit more detail as the Netflix price competition has demonstrated Matrix factorization models are superior to Classic neighbor techniques for producing product recommendations along the incorporation of additional information such as implicit feedback temporal effects and confidence levels that isn't what I'm curious about too like how do you uh use the implicit feedbacks [Music] for the how do you incorporate those signals yeah I mean this is just personalized recommendations are important it seems like it's really important like recommender systems are driving the most amount of value for machine learning for companies I mean like there's a big discrepancy between the number of papers in recommender systems but but uh like from the value that it gives like profit and uh the actual number of papers released I mean I guess it makes sense like nobody is that excited about like ad uh prediction and increasing that but that is fundamentally what striving value uh or like the profit for companies uh and you know it's not only used for that you have a bunch of recommender system but it seems like not that many people are focused specifically on that foreign broadly speaking recommendations are based on one of two strategies uh the content filtering approach creates a profile for each user or product to categorize its nature for example a movie profile could include attributes regarding its genre to participating actors and so forth user profiles might include demographic information the profile allows programs to associate users with matching products of course content-based strategies require Gathering external information so basically like content-based right content filtering approaches use don't use the actual user uh let's see like the the input to the model are based on on data that we have sort of gathered specifically uh for that let's see if we can explain a better way like for example like a movie would be you know what genre it is uh which actors it is and so on like it's it's specific and it doesn't change but for collaborative filtering approaches which is the second one you are actually learning the uh you are learning the preferences of a specific user so in many cases depending on you want to have a mix of them both have their pros and cons oh see I trained music analysts score okay a known successful realization of content filtering is the music Genome Project which is used for Internet Radio Service Pandora a trained music analyst score each song based on hundreds of distinct musical characteristics so basically they like write down the attributes for a particular song which is very you know expensive uh sometimes you know it's automatic you have it but usually someone has labeled it so these attributes are genes capture not only a song's musical identity but also many significant qualities that are relevant to understanding listener music preferences an alternative to content filtering relies only on past user behavior for example previous transactions or product ratings without requiring the creation of explicit profiles this approach is known as collaborative filtering a term coined by the developers of tapestry the first recommender system I didn't know that uh interesting maybe check that paper later on collaborative filtering analyzes relationship between users and interdependencies among products to identify a new user item Association major appeal is that it's domain free let's see well generally more accurate than content based techniques collaborative filtering software what is called the cold start problem yeah so usually I mean if you have the attribute then it's easier to create a solution for the cold start because you're you have those attributes right you have something to to base on uh and you can use some heuristics maybe and someone based on those but if you're doing collaborative filtering you have you know the Matrix is uh empty in the beginning because you have no users and you have no ratings so usually it seems like what a lot of people do is build a content-based solution first and then once they get rating they transition to a collaborative filtering approach or you know a a hybrid system which uses both so it's not like you only have to use one or the other many times you you uh mix them I use both of them the two primary areas are the neighborhood methods and the latent factor models so like for me right uh why I want to learn about this is because like I know but I want to um like actually start to use it doesn't make sense for me to always jump into a deep learning solution it seems like you usually want to have a progressive solution so that you start with something that's really easy and just quick and you can see how it works and then you build on that in many cases you know we know that deep learning works best so when we create a new solution we jump directly to that but creating those baselines you know if you look at kaggle for example a lot of people like they start with the actual like simplest solution like so you have a good Baseline and then you can just progressively iterate and improve on that it doesn't make sense to just jump straight into a deep learning approach so it's important to learn about is what I'm saying uh let's see so for example consider the movie Saving Private Ryan yeah and then also you have two different approaches if you use collaborative filtering for example you can either then look at a user uh yeah like user collaborative filtering which would be you take a user's Vector a representation and then you do a search for like who is the closest user to this one and then look at the movies that that person has liked or maybe you take the top three closest to a particular one and then you compare you know what movies did these people like that are similar to this person and then you recommend those another is you do uh you know that would be the rose I guess like the rose users and the columns will be each item so then you could also do uh an item-based collaborative filtering where you look at uh let's see that would be I don't know like you I guess you take I'm actually not sure I guess you would take the column of a particular user uh or like a movie that they've liked and then you check for the closest ones and then you like take a a mean somehow of movies they haven't liked I'm not exactly sure on that but basically that's the digest of like how you you do it of uh either you do row or column but it's the same approach but one corresponds to looking at which users are the closest another corresponds to which items are the closest so yeah okay so you would take a an item that they haven't rated yeah so that's how you do it so for the column based approach like the item based collaborative filtering you would look at a particular movie they haven't rated you would look at the all users that have rated this one you would look which is the closest movies to this one and then you would take those closest movies and you would go back to that user look at what that user has rated that those movies and take an average of those of top K and then give the that prediction for that particular item that the user hasn't created hopefully that that made sense so yeah those are likely to content collaborative and then you have user and item based all right latent fagmars are an alternative approach that tries to explain the rating say to 20 to 100 factors inferred from the ratings pattern in a sense such factors comprise a computer computerized alternative to the human created song genes for movies the discovered factors might measure obvious Dimensions such as comedy versus drama yeah so you know obviously we don't know exactly what it learns but you could imagine just like embedding so you could learn something useful okay so here they should figure the user-oriented neighborhood method Joe likes three movies uh to make a prediction the system finds similar users who has also like those movies and then determines which other movies they liked okay a simplified illustration of the latent factor approach which characterizes both users and moving using two axes uh okay so let's it's in 2D so let's say that uh we have Sirius and Escapist and we have geared towards female and we have geared towards males okay yeah okay so you would put them somewhere in the embedding space here so latent factor models are based on Matrix factorization in its basic form Matrix factorization characterized as both items and users by vectors of factors inferred from item rating patterns all right cool all right so we have different input data [Music] often placed in a matrix with one dimension representing users and the other dimension representing items of Interest so you know we have ratings for movies like Netflix one to five that would be explicit feedback because the user has you know written down exactly what they like so Netflix collects star ratings for movies uh or you can do like some preferences with thumbs up thumbs down but these are explicit because the user has actually given you a rating but usually you have very few explicit because very few people are interested in like giving their feedback uh usually just people watch something and then they move on one strength of major factorization is that it allows incorporation of additional information so this is one thing that I'm curious about how does it do that all right so how do they do that and why is that a challenge or previous approaches with the nearest Neighbors by treating it as additional features okay so I mean just like putting it into the Matrix somehow incorporating side information Matrix that can also incorporate side information about the users or items oh oh no oh wait what happened all right all right whatever I think I just accidentally removed my by the way I don't know about you guys but I prefer using I think they've sort of abandoned this UI uh it's kind of sad but this UI is so much better than uh like the either the open AI app or the uh yeah because I don't know if they've updated it was a while but like first of all you can use your own prompt you can also adjust the temperature but the thing I use the most is you can go back and you can change you can change your comment here so let's say you ask a question you get some ideas of that and then you want to go back and modify your comment with the previous context here then you can just go back change the you know here um [Music] I'm sorry maybe like can you explain the SVD approach that won the Netflix challenge uh and then it just restarts from there I think that's really useful uh I use that a lot SVD plus plus which incorporates implicit feedback each of the number of movies rated in addition to explicit ratings okay let's find S3 plus plus I wonder if they mentioned SVD plus plus in this paper okay let's see so a basic Matrix factorization Model A major factors in model Maps both users and item to join latent factor space of oh nice I ordered uh I've been waiting for an order that just arrived that uh uh I bought first time I bought some cigars so I'm gonna uh man I'm so excited to to try those out anyways I should mute my phone I get distracted all right Matrix factorization models map both users and item to a joint latent factor space of dimensionality f such that user item interactions are modeled as inner products in that space so basically multiply you know uh that user embedding by that item embedding and then you get if they are very close then um that means that the user probably likes that item Vector Q in each use you associate with a vector again qmp for a given item I the elements of Q measure the extent to which the item possesses those factors positive or negative for a given user U the elements P okay so Q is for the item and P is for the user I guess measure the extent of interest in the user has in items that are high on the corresponding factors the resulting dot product captured interaction between user U and item I uh so as we said like if they're closed then you give a high rating so here Q again was the the user no Q is the item all right so you take the item multiplied by P here okay okay the major challenge is Computing the mapping of each item and users to factor vectors okay after the recommend recommender system completes this mapping it can easily estimate the rating to any item so you know like that's the problem right like we have a very sparse Matrix we don't know what they've rated most of the the ones but doing this Matrix factorization and uh getting those uh like uh embeddings for the user and the item for all items and for all users we can easily just multiply those two and fill in like impute the missing values and then we get um those are approximated ones all right such a model is closely related to singular value decomposition applying SVD in the collaborative filtering domain requires factorize factoring the user a item Matrix all right this often raises difficulty due to the high proportion of missing values conventional svds undefined with knowledge about the Matrix incomplete moreover carelessly addressing only the relatively few known entries is highly prone to overfitting early systems relied on imputation to fill in missing ratings and make the rating Matrix dense however imputation can be very expensive is it significantly increases the amount of data in addition inaccurate imputation might distort the data considerably hence more recent Works suggested modeling directly The observed ratings only uh and by regularizing avoiding overfitting all right so this is what they tried to do this is the objective function like we have the rating we subtract that with the the dot product of the two vectors of the user and the item and then we make sure I guess that the norm of both those vectors don't become too large yeah and then we have two learning algorithms to do that uh stochastic gradient descent Loops through all the that's what we saw I guess in the implementation right that guy had done on Twitter or GitHub I mean sorry uh looping through them that it modifies the parameters and we can look at that here right here look pretty similar to what he had done here so we have we have the biases and we have the this thing Lambda times qi I guess beta is the parameter he uses regularization parameter instead of Lambda and then you have e still yeah so this is the the uh what is it gamma like a gamma is the alpha here but it's basically the same thing right one benefit of the Matrix factorization approach to collaborative filtering is its flexibility in dealing with various data aspects and other application specific requirements this requires accommodations to equation one while staying within the same learning framework okay okay yeah so I guess like it learns uh a bias term where you know uh it might be the case that some yeah some users might rate up like all movies in general like their uh they're optimists so they give five to almost all movies but some people are more pessimistic and they might give three as an average so you know like you know these people who are like when you ask them like how was the movie good uh or something and then they like yeah the movie was okay and like to them okay is that it's good uh but some people are like that movie was amazing uh but some people would never use those words they would just say yeah it was good and if they say that it was good then it was like freaking fantastic so sometimes you know you do this like by yourself you know that some people have a bias of like where their scale is so then you become more sensitive to that so it makes sense that you know you give this to the model as well by this bias what are I don't know how many views viewers there are now but uh what what are uh people what are people doing today and what yeah what time is it for you for me it's uh 11 50. so I'm gonna have lunch soon and I'm gonna go uh do some some work but I wanted to you know try to read a paper at least and make sure that you know learning something today all right so we put uh a biased involved in the rating I guess it also learns yeah I guess like it also learns that some products are just some were just perceived better maybe depending on certain I don't know yeah I don't know exactly how that looks like work like where is the bias how does it look like so we have a rating which is uh okay so previously we only had this part right but now they added bias for the item and bias for the user but what is what is this term okay so they set a bias Okay so you have a I'm not sure what this thing does like you have a bias for the user and the item that's what you want but then they uh do a shorter approximation like this the overall average rating is denoted by mu the parameters B U and B I indicator observed deviations respectively from the average okay so this is the overall average rating for that item no overall average for data set [Music] um now say that the average okay so it's over all movies Okay okay for example suppose that you wanna first order estimate for user Joe's rating for the movie Titanic now see that the average rating over all movies is 3.7 Titanic is better than an average movie so it tends to be rated up on five stars above the average on the other hand Joe is a critical user who tends to rate 0.3 Stars lower than the average thus the estimate for Titanic's rating by Joe would be 3.9 Stars yeah so like I guess here you just go based of a general uh uh bias where you know that in general this person is pessimistic optimistic uh and uh you you know that like it would give 3.9 but based on the knowledge we have about that movie and the actual preference uh this would be in general but like the actual preference for the user to that movie then you would uh include that with this so I guess that'll get make it makes sense otherwise this would also have to learn all the biases uh that every user has and every item has which might be more challenging than just learning the preference so cure I'm curious like did this guy also include the biases yeah so we did this year we have the bias for the user and the item but I wonder did they also have the MU I don't see a mu here oh no it's here so the self.b would be uh the rating where uh just the average rating over all of them it's a pretty good implementation here thinking uh I should probably do the the same right to make a thinking like maybe I should do a video on this because this is like a really really good paper or a highly cited paper I don't know if it's good anymore it's like from 2009 so it's a while ago all right so let's see additional input sources often a system must deal with the cold sore problem uh it's difficult to reach a general conclusion on their taste a way to relieve this problem is to incorporate additional sources of information about the user recommender systems can use implicit feedback to gain Insight uh indeed they can gather behavioral information regardless of the user's willingness to provide explicit ratings so this is uh like how every recommender system does it right uh if you look at YouTube uh you know from the paper review I did on the YouTube recommender system they like mostly predominantly use the watch time of a movie as their signal rather than you know like uh stuff like that which yeah I know like it is definitely a problem in building those recommender system because those companies have a high incentive for profit so and they make profit by making you watch a lot and that's how they make their money so like you as a person like your goal is probably to like maximize I don't know different for every people per every person right but you want to maximize maybe happiness maybe you want to maximize like impact and so on and probably but all of those like your objective function is not to watch the most amount of YouTube videos so that YouTube makes profit right that's not the goal uh but you know there's a correlation between what you watch and what's interesting to you as well that's what YouTube is thinking uh but like the objective function would be give the highest amount of useful information that the user likes while minimizing I get like for me at least minimizing the amount of time that I spend so to me that means I have to like circumvent the algorithm because it just ruins my productivity ruins my happiness level if I'm just watching like shorts all the time like I don't think people are happy you're watching Tick Tock so like a really big problem with every recommender system is that the objective function they create are very different from what the people want basically anyways that was a rant all right a retailer can use its customer purchase or browsing to learn Tendencies for Simplicity consider I mean this is also the the one that Netflix is using like they they look what movies are you watching and they give you more of those movies to watch and uh yeah I guess like it's a it is a good somewhat good objective function at least I knew the notes a set of items I expressed an implicit preference this way the system profiles users through the items they implicitly preferred here a new set of item factors are necessary where item I is associated with x i okay okay so you have like a a latent vector or like a vector for every item which is a size f and you basically uh if you take the mean of all those another information source is known user attribute for example demographics again for example Simplicity consider Boolean attributes where user U corresponds to set which can describe gender age group zip code income level and so on a distinct Factor Vector corresponds to each attribute to describe a user through this set of user Associated attribute okay and then the Matrix factorization should integrate all signals well let's see so you have uh okay so you have like the okay so instead of just adding for the particular user here you also include the normalized average of these and this one so you basically just add the information so that's how they incorporate implicit feedbacks I was confused of like how they did that but that kind of makes sense so I'm curious here what what is the x i what does The X I actually mean item I is associated with x i okay so like is x i the uh just the uh the vector for that item I guess so they just take an average of them like so then we have the feedback of what they implicitly liked but what is this one then so you have y a uh a u h you corresponds to this set of attribute attributes a u which can describe gender age group zip code income level and so on a distinct Factor Vector corresponds to each attribute I'm a bit confused over that I wonder if GPT can help us out Let's uh this is such a popular period probably it knows about it um with the implicit feedback this is actually insane that it knows this and it's pretty good um I'm confused over the term uh I think by the way like Khan Academy uh uh what is it called Salma Salman Khan like the the founder of it when saying that these GPT models will be used as personal tutors I'm already using it for that like all the time you can't trust it completely but you can't trust anyone so I mean like you you uh you have to be careful but in general like you can definitely trust it in general uh so what are we confused about we're confused over the term uh YJ YJ I'm not sure not sure exactly what that term entails but no it's not the no he's missing a term right yeah no this is not right actually um it seems you're missing a charm um in equation six in the paper there are two sums one One X I and one yeah so you can't trust it but basically it's pretty good still wrong okay just a bit challenging I guess I think basically this is the items that they've won and then this is uh the attributes for a user I'm just not sure exactly what this uh what the f yeah like I'm just not sure what we're summing here so here we're swimming over all of the ones that they have uh had implicit feedback for so we're averaging those but here we are uh for some Vector of the user uh attributes of some kind what that means I don't know all right but then they also introduced temporal Dynamics they have been static but product perception popularity constantly change as new selections emerge similarly Customs inclinations evolve leading them to redefine their taste thus this system should account for the temporal effects reflecting the dynamic time drifting nature of user item interactions that's actually an interesting uh okay decomposing ratings into distinct terms allow the system to treat different temporal aspects separately specifically the following terms vary item bias as user biases all right so they basically include a time there and user preferences uh the first important effect addresses the fact that an item's popularity might change over time uh and such as an actor's appearance in a new movie therefore the models treat the item as a function over time I mean it makes sense actually so basically we update the equation to have a user over time uh user bias over time item over time I wonder if this should also be over time like the average rating of movies uh change over time as well but maybe you can see that as static I'm like not too interested in this temporal aspect I'm a little bit curious of how they did that but not in uh really interesting movie all right so they also have confidence so it can use higher weight to more confident ones not exactly sure what the confidence is okay so we have confidence scores on how much they would like something all right so they have the temporal aspect but then also the varying confidence levels all right so here they go through their actual like solution thing um okay uh factorizing the Netflix usually moving Matrix obviously discovered the most descriptive dimensions for predicting movie preferences we can identify the first few most important dimension in movies location okay so basically the plain one has this with biases they improve slightly with implicit feedback they improve a lot because like there's a lot more information there and then they introduce the temporal uh Dynamics uh and that improved a bit as well not that much though seems like the biggest Improvement was I guess it helped as much maybe okay so that's interesting yeah okay I mean all right major infections and techniques have become dominant methodology methodology within collaborative filtering recommenders experience with data sets such as the Netflix price data has shown that they deliver accuracies period to classical nearest neighbor techniques I'm curious how the classical nearest neighbor techniques worked at the same time they offer a compact memory efficient model that system can learn relatively easily what makes these techniques even more convenient is that the model can integrate more uh yeah more forms of feedback I uh it seems to me that they didn't touch that much unlike the cons of this uh seems like like the the scaling problem uh I mean they talked a little bit about the cold star problem maybe that it's difficult but they didn't talk that much about scaling maybe that is isn't an issue I'm not sure but yeah I don't know how many viewers there are in this one I just thought I'll make a a short stream going through a paper uh and that was it all right I think I'll probably end it here I don't know how long we stream but [Music] um yeah like one hour pretty decent to go through a paper maybe I'll make like a implementation video on that Matrix factorization thing uh maybe on like movie lens data set or something and we can maybe like start to build a repository of a pro like a recommender system approaches and try to understand them from scratch all right let me know if there's any uh questions otherwise thank you for joining the the stream and uh yeah hope you have a good day or I don't know what time it is good day good good night whatever time it is for you uh do you have any plans on next stream uh yeah I mean like I I uh this was pretty unplanned to be honest I just figured you know hey I'm gonna read this paper why not do it on stream might be interesting for people to to join in uh uh yeah I might do another one [Music] like one thing that I've done in the past is like like streaming doesn't need to be a big thing uh it can just be hey like now I'm live I'm doing this and uh I want to do it more frequently uh just in the past I've kind of become a little bit distressed of streaming but I think that's just something that I need to get used to because like you will make mistakes and you will kind of look at them sometimes on stream and that's fine uh it's not going to be as refined and as a proper video but on the other hand people are not expecting that either so so yeah I might do another one live tomorrow uh if I continue doing this maybe I'll put a schedule I don't know all right guys I'll end the stream uh thanks for joining and I hope to see you uh next video or in the next stream bye guys