Niche vs Mainstream

Key Takeaways

Welcome to Data Skeptic, a podcast [music] exploring the methods, use cases, and consequences of recommener systems. Welcome to another installment of data skeptic recommener systems. Today, we're going to start entertaining the idea of decoupled recommener systems. What do I mean by that? Today, pretty much any recommener system you're going to encounter is an in-house thing. If you go to Netflix, the recommendations you get there are from the algorithm produced by people who work at Netflix. But what if there could be a marketplace there the way your phone has an app store? There could be an algorithm store. Of course, that would require these platforms to open up, but perhaps there's a very good reason they should do so. Towards that end, today we're going to discuss the simulator for modular recommendation ecosystem or s'mores framework. We get deeper into these decoupling ideas and explore how s'mores can be an empirical tool for studying how ideas might play out in that space. My name is Anns Bhay. I am a student at the University of Colorado Bouldered by Dr. Robin Burke. I'm at the information science department at that recommener systems lab and most of our work in this lab focuses around multistakeholder fairness and recommener systems. Yeah. And basically the work I do is mainly around evaluating alternative recommener systems. >> You use the term multistakeholder. Could we talk about who those stakeholders are? >> Yeah. So usually whenever we think about a multistakeholder recommener system, we think about recommener systems that work as matchmakers. So think about having someone who's providing a service. So for instance, if we're talking about Spotify, that can be the artist or think about a YouTube that can be like a content creator, Door Dash, the restaurant or the driver, someone who's receiving the service. So the one who's providing the service we call them providers and the person who's receiving the service we call them consumers and that's the user that we usually call that the user uh on everyday basis and that's the person who's like listening to the music making the order on Door Dash. We have the platform which is something that we cannot ignore because there's a lot of conversation like oh we need to do the matchmaking but we forget that the the platform also need to you know like make money to keep running. These are the three stakeholders we talk about and most of the time when you see in the literature when whenever we hear user we don't think of providers as users for some reason but they are as valuable re uh users in the uh in this whole ecosystem. So when you consider fairness, is it for the provider, the consumer or the platform? Or maybe all of the above? >> All of the above. Yeah. You need to think about everyone on the platform because recommendation as an ecosystem needs everybody to be happy to some extent to stay on the platform. So let's say if I join a platform as a provider, let's say a content creator and my content is not reaching anybody, then I'm going to leave that platform. And then you would have like a platform with only like three or four content creators who just like make the content. So like how do you balance between all of these multi-objectives on the platform? >> What would it mean for a recommener system to be unfair? >> So the the fairness issues in recommener systems are similar to like machine learning as a whole. So you have two fairness issues that you would usually find in the literature which are representative fairness and allocative fairness. So let's start with the first one. Let's say representative fairness and talk about it in the the idea of like recommended system. So let's say you have a job recommendations and you're recommending to females uh like a less paying jobs or like a specific stereotype of jobs and you're presenting to males a different kind of job. So that's as a as a receiving end. Also another way it could be like I would jump on a search image and I look up I don't know an Arab man or like an Arab man. So like what kind of photo would you see whenever you look up that because that is in itself is a retrieval. So it's a recommendation. So that's the representative part. The other part is the allocative awareness which is let's say we're talking about like content creators for instance. So the idea here me as a new content creator joining the platform will I get any chance to be recommended or only like the very famous people on that platform will be presented to the consumers. So these are kind of like the two main things we talk about whenever we talk about fairness and machine learning. >> Well I know you probably can't have surveyed all of industry and all of implementations. There's a lot of recommener systems out there but I assume most of these are written inhouse. Is that true? >> As like the recommener systems. >> Yeah. So like at Spotify or YouTube, whatever they're doing, it's probably a trade secret by employees inside the company. [clears throat] >> Yeah. Exactly. I mean the thing about recommener systems as a whole, they are multi-stage. So there are like two narratives. Uh the first narrative people think about recommener system as just like one algorithm. You just like put that algorithm and then you get a specific output. But in reality that it's multi-stages. So for instance, we're talking about an enormous platform like YouTube, you cannot put every item like those millions of videos or maybe I don't know billions at this point. You would put them in the algorithm and they would spit out recommendations for a specific user because that's that's so many. So you have different stages. So you have the retrieval which is you would get a a portion of those items and then you would put them in an algorithm and then that algorithm would focus on the relevance. Let's say we picked a thousand item. We rank them by the relevance to the user and then we do another stage which is the um filtering. So we filter whatever items that we don't want to show to the users and then we do reanking. So most of this fairness happens at the last re-ranking stage. However, if any let's say bias or unfairness happens in the previous stages, of course, whatever you fix at the end is not going to fix it. So let's say if I'm not picking the niche user from the beginning and my like first 10,000 item, that niche user is not going to make it to the end whenever I do like some sort of reranking. And that's where you see like most of the responsible AI work is happening in the re-ranking phase because that's when you have the opportunity of like okay let me add users from this specific group let me put these users more on top of the list or something like that. >> Well reanking makes sense it's intuitive. Are there other ways to approach the challenge? >> That is where we come with our idea. So we were like okay we we have the the reankering but what if we just make a new recommener system? What if we let people create or like algorithm developers have their own algorithms that they would attach to this platform and then you would have like a curated algorithm and by this plurality of algorithms by having like more algorithms we might curate to different users and also different providers may make it to the consumer at the end. Is that the s'mores framework we started talking about? >> Yeah, the the idea there comes from the uh the algorithm store. The algorithm store have like two people started talking about it. There's like this manifesto which is called like the three-legged stool and the idea comes there from like what if for instance on an platform like mastadon like a federated social media platform what if you let users create their own algorithms and then they would create their own needs. So you would have like for instance a user would choose from like a spectrum of algorithms. You're like oh I want this one that for instance gives content about this specific niche sport and I don't want my niche sport to be within the like this big mega sports algorithm. So I'm going to choose that one. And also you can think about it as like uh an algorithm that fits my values. So if I am on Door Dash, I might want an algorithm with the restaurants with only like ethical meats or like kosher meat or something like that. So whenever we have this idea of like algorithm pluralism, we would have more opportunities for users to choose the path that they want. And there is this another idea by Fukayama which is the middleware that is basically just the idea of like middleware and software engineering. So you would have this software that would work alongside the system and it is more of like a modular component where you can replace one algorithm with another algorithm. So this idea of the friendly neighborhood algorithm store is interesting. It's the opportunity for researchers like yourself to come up with a new idea and put it out there. within the platform. I don't know, do like a multi-arm bandit thing or or pick their favorites and experiment with them or is your vision more that the user has a choice here? >> So, we see it as the user has a choice. The closest example here is blue sky. So, blue sky has this idea which is I think started in like the past couple of years uh which is the customized feeds. So, you can think about it the same way. So the user would try to look up a specific feed or an algorithm. There should be like specific metadata about that algorithm and how it works. And then the user can choose their own algorithm because what we're trying to do is to listen the control of the main platform that owns the item. Let's say we're trying to listen to control of like what's trying to surface and give that control to the algorithm designer because we don't want this centralized power of like only this platform can decide what the user sees. We want more people to decide. >> I'm curious about you know being able to have that choice be empowered to pick my algorithm but it's also another step I have to take. So what are you actually trying to measure in your experiments? That's a really good question and that is the the question of the the user burden. So like whenever we we add more algorithms, we're asking the users to investigate and see their own algorithm and like find the one that serves them and that's and we try to see it more of like a a subscription model. So like for instance you choose whatever newspaper that you want to read or like whatever uh streaming service that you want to use and it's this the same idea. So we started with this research question. Okay, if we have multiple AL algorithms instead of one, would that be beneficial to a specific group of users and or like the the users who are interested in that specific content? Because there's no platform out there to try these ideas. We had to build our own environment. And we were like, okay, what is the best way to do this? And that's when we came up with the idea of what if we just make a simulation environment. So we're just going to simulate this whole basically ecosystem where we have providers and we have the platforms and we have the consumers. So basically we use real data. So there are like multiple data sets uh out there that recommend system researchers and machine learning researchers use. So for instance one of the famous one is called movie lens and that is user ratings data set like so basically user rated a lot of movies and what we did there is that we took all of the movies like that are on the data set we found their studios from a different data set and then we were like okay these are the providers so the provider is the studio in this ecosystem and then we already had the users So we used those users and then the last part which is the platform and we're we basically because we're thinking about this idea of like the algorithm store we had to simplify it so we can measure what does that change. So we just created two recommenders. So like the store has two recommenders at this point. So it has one mainstream recommener and one niche recommener. The niche recommener basically focuses on a specific genre of movies and the mainstream would just see whatever like other users on the platform wants to see. So basically we used an algorithm called SPD. Yeah, basically it's just like a one of the the popular recommendation algorithms. We ran the experiment and we had a multiple switching mechanisms. So we had a utility measurement which basically says if the user does not see the genres that they want to see in a recommendation list then the utility would decrease and if they see the genres that they want to see in that recommendation list then the utility would increase. And we had a specific threshold and we said if the user's utility would drop below that threshold then the user would go and choose a different uh recommener. So we had this idea of like a switching mechanism. By the end of the experiment, we found that users who moved to the recommener that tailored more to their needs would have a higher [clears throat] utility and also the providers who did not make it in the mainstream recommener because they were a niche provider. When you have a a recommener that focuses on their content, then it's really easy for those users to find that content instead of trying to find it in this abundance of content on the mainstream recommener. [music] How many login does it take to answer one business question? ERP for orders, CRM for customers, HR for headcount, accounting for spend. Then someone sends you a spreadsheet that contradicts all of those other systems. Scaler ends that chaos. It unifies every business system, structured and unstructured data into one governing source of truth. Ask questions in plain English. Get instant, accurate answers. No exports, no chasing people across tools. No more three plus platform treasure hunts just to close a simple report. Scaler works with Snowflake, Datab Bricks, and every other data provider you already trust. No migrations are required. It understands what your data means, where it lives, and gives you access 10x faster, saving you time and money. Go to scalar.com to get your free demo today. That's s c.com. Once again, scalar scr.com. get a free demo of Scaler and find and finally unify your business data once and for all. So you mentioned having two recommenders obviously the ones for mainstream and then one is niche you said for a specific genre. Was there one genre you picked or did you have several niche genre models? >> So we tried with different data sets for the paper that we published we had horror for uh movie lens. select the horror genre and for we used an umbar data set which is a music data set and for that one we used soul funk for each data set we tried one niche ideally in a simulation environment and that's the work that we're working on right now you don't want just two you want this idea of niche to expand so let's say users who go below a threshold and then you try to create a niche from that group of users below the threshold and then until you reach x amount of recommenders. So [clears throat] having these strict simulation environments are really good in a sense of like they would give you the the first picture like oh this is a potential thing but I think there's a lot of work needs to be done to come to the idea of like oh this is the thing that we need to do and these are like the clear tradeoffs of this work. Well, two different recommenders can differ in their training data, in their specific algorithm, or maybe just some parameters to the algorithm. How did your models differ? >> For both of the recommenders, we wanted to be consistent in terms of their parameters. However, the data is what's different. So, the niche recommener only recommends items that would have the genre that we defined. So let's say if it's we call it like the horror recommener then we'd only recommend movies that has the horror genre. The mainstream would basically recommend everything. The data is also split. So as a simulation environment the users would arrive. So let's say like the first day of the simulation that's where the users are going to interact with the recommener system. But before that the recommener system does not know [clears throat] anything about the user. So we're trying to simulate an environment that the recommener sees every user as a cold start case. So that's a new user. So because we start with a mainstream recommener, then the mainstream recommener would build profiles on the users and then we'd use the data that the user selected as a training data for that user. And then when the user moves to the other recommener, the niche recommener does not know anything about the user at that point because it's also facing the same problem. If it's a cold start, this is a new user that is arriving. So we have this separation of data between the two recommenders. And that is a question that we're also interested in which is should users transfer their data like if I'm moving from one recommener to the other recommener should I take my data with me and there is a research that is evolving from the GDPR and some other legislations which is the idea of data portability. So me as a person who's moving between recommenders, should I take my data with me and how would that be even beneficial to the other platform if I take my data with me and what are also like the tradeoffs there >> in your simulation the users are going to follow that switching mechanism you described earlier. Do you have any highlevel observations about like how often they switched or you know things along those lines? Yeah. So the users or we call them like the niche users which are the user who are interested in like the horror genre for movie lens or the soul funk for the armp users I would say like 90% of them would switch to the recommener really early on like the niche recommener and then only few of them would keep switching back and forth and we had this idea of like back and forth switching because Some users would not find any of these two recommenders satisfying. So let's say we can group the users to like three three groups. We have the people who have like the action and the drama. Those people are happy like in their mainstream recommended. They're always going to get the the action and drama. There's a an abundance there. And we also have our niche recommenders who are like really interested in the horror. So those would move to the the niche recommener and they will be happy there. And then we have a group of user who would not be happy in these two. So these this group of user will be like switching to the generic recommener. They would get good recommendations then it's going to get lower. Then they're going to switch to the other one. And that's when you need a third recommener to kind of like solve this problem. But uh you don't want to keep increasing recommenders until every user will get their own recommener. Right? So there's like a threshold there that you need to find where okay until what point I can add recommenders while I'm adding value to the users like at what point does that stop adding value let's say? >> Well the more traditional approach is just the monolithic recommener. They would have the mainstream one and that's it. How do you measure the improvement you get by adding in the option for the niche recommener? >> Yeah, so we basically measure the utility for every provider. So for the providers, we measure their utility for the generic providers and then we measure the utility for the niche providers. And for the consumers, we do the same thing. We measure for the the generic and the niche. And what we found that for the generic providers they would use some utility because basically some of the like the views or the clicks that they were getting they were getting from niche consumers and now those clicks are moving somewhere else. So like those niche consumers are going to click on like the niche providers. For the niche providers, they basically gain so much utility because they have a recommener for themselves and users. Whenever they go there, they be like, "Oh, this is like like the niche content that I'm interested in." So, every niche provider will get some sort of uh exposure on that recommener. As for the uh consumers, we also saw the same thing. So, like niche consumers would have a higher utility. However, then the generic consumers sometimes when they switch to the niche recommener, they only see stuff that they are not interested in. So if if the consumer makes the switch to the other recommener, then their utility might drop. But we also found some users who we did not label as niche, but they stayed on the niche recommener because they liked it enough to stay there or like they liked it more than the mainstream one. So there's definitely a tradeoff to make there. And for the provider tradeoff, I can argue that for those mainstream providers, like those clicks weren't supposed to be yours anyway, so they should go like to the niche. But for the consumer side, that's the one that you don't want. You don't want uh if I'm like interested in like comedy, uh don't move me to a horror recommener. That does not make any sense. It's intuitive to me that a niche, you know, horror lover who gets the niche recommener is more likely to enjoy it. And I see your point about there's a marginal cost. Those views went away from maybe mainstream providers. If there were a big platform, let's say it's YouTube or Spotify or whoever who was interested in this idea, but the product manager says, "Well, I can't afford those costs." How do you look at it practically? >> Yeah, that's a that's a really good question. So I think because of the simulation environment that we have we are facing those costs. I don't think that those costs are going to translate directly to like those platforms because one thing for instance that we did not account for for our simulation environment is the say like the name of that recommener. Now, for us as the creators, we know that that's a horror niche recommener. But in the the switching, the the user does not know that it's a a horror recommener. They just think of it's like, oh, this is a different recommener that that I need to check out. So, let's say for thinking about a platform like YouTube, I like playing guitar, so I'm interested in like guitar content. So if someone created a recommener for guitar tutorials or like more specific like guitar tutorials with tabs or something like that, then I as a user I know that this recommener is for this specific thing. So that kind of like lowers this cost of the user seeing stuff that they don't like. And I would argue that me as someone who's like going to YouTube and seeing like everything on the same page, that's when I'm losing utility. Uh, and I would gain it more if I can choose what I want to see actually. >> Are there any technical barriers to getting an idea like this that there are third party algorithms on a platform? Are there technical barriers to making that a reality? There are some side effects of a recommender algorithm. So for instance, we have the filter bubbles where everybody's seeing the same thing. So like for instance, if you are either on the on social media and you're either like on the right or the left, you'd only see content that is from that political spectrum. And that's what we call like a filter bubble. If you let users design their own algorithms, they might even exacerbate like that idea of like, oh, I'm just going to focus on this specific content. I'm not going to allow for any content from like the opposing party or like opposing ideology or even like slightly outside of that ideology. when uh Fukuyama introduced the ADF middleware, he said that a platform oversight might be helpful there. So the the platform itself need to have a specific requirement for someone to create an algorithm. Another thing can be like a transparency. So we need to have more transparent algorithm development like can the users for instance see this third party algorithm and like how it's developed and not everybody's going going to understand like the technical part of that but we know it's open for anybody who is interested as for like the technical development side because we see this idea of as like the platform has the storage of the content and these third party algorithms just have access to that storage. So like development wise, it's not a very hard idea to develop. It's more about like all of these tradeoffs that you need to consider like as we said like the user privacy like is [snorts] every third party algorithm going to own the data is the main platform owning the data and then these third party would have like a specific access. So there are a lot of things there that we like a platform needs to think about to make this happen. >> So what is the incentive for a platform to adopt third party algorithms? I would say the main incentive would be that users would have a better experience on the platform as they would find the content that they would actually interested in because I think there's also like different kinds of platforms. There's there are the platforms who focuses mainly on let me just like put more ads in front of the users. I want to surface specific material in front of the users. For those platforms probably this is not what they're looking for. But with the platforms do actually want the user to have a good experience on their platform and they're not just trying to like go over engagement and these kind of things. Those platforms basically can benefit because they are opening a new market where other ideas can come in and then those ideas might be bit more beneficial to the users. So again like back to this idea of me liking guitar. So a recommener that focuses only on that that would be very helpful for me because like when I think about YouTube there is what I want to see and there's like what the recommener is giving me. So what I want to see is more the things that are aligned with my interest but the recommener is like oh you like this cat video so you're only getting cat videos from now on. So this kind of constraints of a third party algorithm can help the user also choose what they want then the user would be happier also being on a specific platform. >> From your perspective is there a trend in industry to move in this direction or we maybe just at the talking about it phase? >> I think because blue sky had something similar. So at least we know that there is one part of the industry that is aiming it like going in that direction and it's very promising because they have this idea of like created feeds where users can create their own feed and add specific filters and then you have a feed basically uh with a specific sort of content and I think also there are some like startups that are focusing on this idea with like building feeds for blue sky. Um I don't remember the name of the the one the very famous one but basically there are some companies who are helping users who are interested in creating their own feeds. I think Spotify when they added this idea of like you create your own playlist and even like YouTube and then you can share your playlist that is one version of it. That is one version of me like creating tailoring content to other users. I would argue that Spotify's AI prompted stuff is like taken over, but a lot of like YouTube playlists are still very popular, I would say. >> So, the s'mores framework is available on your GitHub page. People can test it out or extend it or uh who knows what else. >> Mhm. >> Tell me a little bit about that choice and about what users will find there. Yeah. So we uh we've been working on this simulation environment for almost like two years now and just like tweaking stuff, adding stuff. It is on GitHub on our like that recommended systems lab repo and once you go there you can just clone it. There's information there about how to install it. We have also other repos for experiments. So if you want to see an example of uh how to run an experiment, we we we also publish the uh the experiment repos because we we are separating of like this is the simulation environment and we have other repos for simulations like the experimentations that we use for our papers and those usually would have like our results and our inputs. Yeah, it is. It is available and we did not talk about like contributions yet, but I'm sure that will happen at that specific point. >> Very cool. Yeah. Well, what's next for you? >> So, I am really interested in this idea of alternative recommener systems and user choice. The next study that I am interested in is okay. So, we gave users their own algorithms. Let's say we created this niche algorithm and users join that niche algorithm. How can we allow those users on the platform control what is on the algorithm? So we we go more to the another form of user agency on the the reranking stage and mainly thinking about it as recommener systems like the the group recommener uh lens. So if you have let's say even like a [clears throat] Mastadon instance with a thousand or 2,000 users, how would those users decide what their recommenders is going to look like? Or like if you have a subreddit, how would those people on your subreddit, if if they have the chance, how would they choose what's allowed on their feed, how it's ranked, what is important to them? So, moving more to the uh the agency part. >> Makes sense. Yeah. And is there anywhere listeners can follow you online? >> I'm usually active on LinkedIn. So, you can type my name, Anna Spai, on LinkedIn. And I don't know if there's anyone out there with the same spelling of my name. So, [laughter] you're probably going to find me. >> Lucky in that regard. Yeah. >> Well, thank you so much for taking the time to come on and share your work. >> Absolutely. Thank you, Kyle. Thank you for having me. [music]