Russian Election Interference Effectiveness

Key Takeaways

foreign there has been a lot of discussion about the Russian interference in the 2016 election and without a doubt they did make those efforts to interfere to me the more interesting question is how effective were they that's a very difficult question to answer but we're going to do our best discussing today's paper on micro targeting socially divisive ads a case study of Russia linked ad campaigns on Facebook I am a researcher at Microsoft research in the Montreal lab I work closely with the group called fairness accountability and transparency and ethics in AI or which is also called as fate and that is very relevant to the works related to ethics and responsible Ai and fairness and transparency related questions of machine learning and AI systems in the real world so that's the group's interest lag and general interests of mine I did my PhD from Georgia Tech last year and my research has been in Social Computing computational social science and human centered machine learning where my primary problems have been around the question of measuring and studying well-being and Welding specifically related to workplaces and college campuses where which I also call as situated communities the communities which are geographically co-located at Microsoft research I'm also interested in problems related to how these measurements have challenges like they could also cause harms what are the concerns about these measurements and these AI systems in the real world so that's what my research interest is like so like there are measurements it's a passive sensing and other sorts of data like we live in the in the digital age like there is so many forms of data the studies have been with measuring different behaviors and well-being through this data but also there are concerns there are challenges there are ethics related to it and their privacy concerns as well so how do we mitigate these concerns and still realize the benefits uh kind of my research also lies at the translation between Theory to practice so there are like potentials and Promises in the theory in research but if you were to take it in practice there could be new challenges so how do we realize these challenges and kind of mitigate or prevent this cons like concerns and harms happening if these systems were to come into practice well you mentioned a term that I'm starting to hear more and more often and that's human-centered machine learning could you share some thoughts on uh why we need that specific subset and how one can focus in human-centered ml over just generic ml this is a term that I did not coin up but it has been uh going on for the last few years this kind of also draws interest from a lot of like user-centered research and HCI is one of the communities where I kind of frequently publish in and kind of engage a lot with so HCI is about human computer interaction and one of the core components of HCI is the human or the user is at the primary focus so basically if you were to think of it this way there would be one one style of approaching a problem which is about data driven style which is more like we have a data and we we want to build better models we want to optimize for better accuracies and better models how do we build better machine learning models so that is one style of approach the other approach would be about which is more problem-centered like humans are individuals are having this particular problem in this in social science space or in this world real world how do we kind of improve the lives of this uh of the individuals or what we can assess about these individuals what we can study about these individuals that we would not be able to study through other mechanisms so in if we are studying this kind of I mean that's why like if you think of it this way the problem is very human-centered in itself and if we are adopting machine learning approaches to kind of study these problems that's when it becomes human-centered machine learning one example would be understanding well-being I mean that's what I gave an example as traditionally well-being has been measured through survey instruments like the individuals where people are given survey instruments and they respond to survey questions which are very psychologically validated but the problem with that uh one of the limitations of surveys are that surveys are not timely service can only be done in snapshots so like you cannot have like a rich and like every point of day kind of data so I think that's when passive sensing based approaches have come up like you have like say for example smartphones and wearables you kind of get the data throughout your throughout the day so that data could be used to also measure people's well-being or people's step counts so for example our variables can measure our step counts uh or heart rate so that kind of real-time data is kind of coming up so if you were to kind of think of the machine learning approach that kind of measures the step counter uh the heart rate it's kind of human centered in a way that it's about the individual uh what it is measuring something about the individual and there are like other ways to think about it like personalized and person-centered approaches as well but yeah so that's the kind of a broad gist with an example so as I mentioned like my my primary interest has been with social Computing and computational social science social media I mean the way I see it so I was giving an exam giving some examples of survey based measurements I also gave the examples of passive sensing as sensors and variables and smartphones social media the way how I see it is also a form of passive sensor why I call it a passive sensor is that people provide naturalistic and self-motivated data on social media social media platforms are used to engage with and share with others as researchers this data or researchers or clinicians or practitioners this data has been also useful to measure people's behaviors and well-being like for example our language how I speak has its own strengths and can be used to measure people's sentiment emotions that's how a lot of these fields started and also now with the other constructs such as stress depression anxiety and what not I mean that has been also one of my research interests when I'm talking about people's language it's also very much theoretically grounded like for example in the real world it's irrespective of sensors and social media when we go to psychologists or counselors the thing that happens is counselors kind of see how we are speaking or what we are speaking and that is kind of what translates to what happens on social media like how we are talking to uh one another what kind of language we're using kind of does our language use keywords or phrases which can be indicative of someone going through um suicide with ideation so can we intervene in that case and support and provide them you know support I mean there has also been research which shows that how people are getting benefits on social media platforms with respect to getting support emotional support informational support or feeling better with their mental health and well-being sort of a process so this has primarily been my research interest and as I was mentioning I was I also focused on college students and college campuses and college students social media is also a key platform for young demographic and college student demographics so it's very ubiquitous and widely used and the way I would also characterize them is that online communities of college students is sort of like analog for their offline College communities and in my research I've studied how say stress levels of students vary when there is a crisis on campus like a gun violence event or some other event or a student that event and how students react to that with all those impacts it's clear social media isn't this passive thing we kind of do on the side it's fully integrated into our lives or for many of us and we can measure quite a bit about people so it also stands to reason social media is a good place to go and try and Influence People yes and um so I mean and also like when we're talking about social media social media has been evolving significantly over the last few years like like platforms such as Instagram and Twitter and whatnot are coming or SnapChat are like super prevalent nowadays compared to how they were like five years back I mean like Instagram reels or Tick Tock and one of the ways to influence I mean this also started like over a decade back so with Twitter and there has been research about who are influencers on Twitter and they necessarily do not have to be celebrities but basically if there is a Content or something that is shared on social media it has an audience and people if they feel engaged they have with a retweet or re-share that content with others and we have also seen the bludgeoning body of research about how these platforms have spread so both misinformation as well as fake news sort of influencing sort of thing and and offlet with say the kovid and pandemic we have seen different kinds of information overload on so on social media we don't know what to trust what to not trust and I'm not just thinking about offensive or trolling the kind of content like those anti-social content people have been platforms have been trying to moderate as much as possible but sometimes it's not necessarily anti-social content people share content which necessarily may not have an intent like an evil intent with that but basically people probably genuinely believe and they just keep on sharing it and the reason also I also mentioned about social media how it has changed is that platform such as WhatsApp I mean they're like so many WhatsApp groups and how people have been sharing many misinformation or a kind of provocative content about different sort of things it has also led to different sort of instigation and incination of people with different ideological groups I mean when I'm speaking a political space that's also one way how people have responded and how people have been influenced in this platforms and also specifically when we're talking about this particular paper about ads targeting that has been one way that the social media platforms like when I'm talking about Facebook Twitter uh these platforms also facilitate individuals or groups or I don't know institutions to share ads about their content and based on that their content gets boosted they can kind of share that like can have more cascading effects and one way to share ads is targeted advertising and now speaking of targeted advertising is sort of different from conventional advertising conventionally how we would see ads is like through newspapers or through TV networks and each channels will probably show different sort of ads and basically if someone if anybody watches a channel and whoever reads a particular newspaper would kind of get exposed to the same kinds of ads so the targeting in those kind of like conventional traditional forms of advertising was through the channels or the newspapers themselves but now these platforms are providing opportunities for targeted audience like we can provide specific attributes like people who live in this particular area who have this kind of Interest Who can belong to these racial or gender groups they are the ones who I want to Target and where while on this surface it has many advantages like for example a local company which wants to sell sell its product they can have a very specific audience they can invest in a very specific investment advertising in a very specific audience that would probably be likely to be engaged with their product or or buy their product or come to their shop or that kind of things but when these ads are also used for political reasons like when we're talking about 2016 ad campaigns there was a lot of ads on Facebook and one thing that we've studied in this work was what happens if the ads are micro targeted like for example if an ad I mean I'm just giving a very specific example and we can go into more details of the people later but I can give a very specific example say an ad which is about a fake news which is about say the Democrats say Hillary Clinton is a devil I mean I'm just making this up and this is not a particular specific example but say this is a particular example like this is an ad now what if this ad was only shown to people who are very dry cleaning I mean people who have interest in who are conservatives and conservative leaning multiple things that might happen is that this ad which is a probably a fake news a false false claim which is probably should have been reported on the platform like if it exists on a platform it has some instigating content in it but if it is only targeted to very micro targeted audience of people who are right-leaning people who are kind of like very conservative they are not likely to report this ad and if that happens this kind of content will keep on staying on the platform and they will never get noticed and this kind of content will never be moderated so I think that was a strategy that happened with respect to micro targeting of advertisements in the 2016 elections which we specifically study in this paper [Music] estrato is business intelligence that helps you get back to business as a progressive data leader you need to be sure that you and your team have the best resources having the right tools makes it quicker and easier for you to get the right insights at the right time Estrada was a modern bi solution brought to you by the experts at vizlib purpose built for the cloud estrado is the next Generation solution for intelligent accessible data analytics with Estrada's approachable no code solution you can do more than just Empower your team you'll be able to facilitate quick and easy adoption across your entire organization and ensure that everyone in every department has what they need to answer critical business questions your team can enjoy unprecedented opportunities for invaluable collaboration with Estrada's intuitive creative experience and they're integrated right back functionality enables you to write edit and comment on your data without ever leaving your business app with live query you can explore or Big Data instantly and enjoy greater agility performance and security best of all you can try a strato for free visit estrato.io data skeptic to sign up they've got a collection of example workbooks you can try in the cloud or get started with your own data visit estratoastrato estrato dot IO slash data skeptic thanks to our sponsor BB edit from Bare Bones software if you're a Mac User and you have any need whatsoever for a text editor which let's face it that's got to be all of us you have got to check out bbedit bbedit 14 provides Mac users the most powerful professional grade text editor around it's known as capable and rock solid and can process files that are large enough to bring other editors to their knees this tool has been beloved for over 25 years and I can understand why if I'm just typing up some simple hello world I guess any text editor will do but when I really get into the weeds and need to do some data processing maybe I've got a big messy file I need to find all the non-ascii characters and remove them BB edits there for me it has best of breed multi-file search and replace and great scripting support for text Transformations there's a whole menu of text Transformations that you'll find Handy if you're a serious data scientist or software engineer you're going to bump into situations where you need a powerful text editor so download and try bbedit today you can get it at www.bearerbones.com see how much your productivity can increase barebones.com and download BB edit well if I were to try and make I guess the steel man argument for why micro targeting is good I would probably point out that you can bring the issues at of concern to different groups to that group you know in any elected official election maybe there's 40 positions they hold 20 of them I don't care about so if you advertise you know that they're pro-chooing gum or not it doesn't impact me I would like to know the things that are more likely to impact me so I don't necessarily want to ban micro targeting but as you pointed out there's this consequence of it's kind of makes things a bit hidden where do we meet in the middle absolutely so that's a great example and also other other that I didn't talk about in micro targeting micro targeting has also been used to discriminate against particular racial or gender groups this group of research is also kind of influenced Facebook to kind of get rid of certain parameters or attributes in micro targeting I think at the beginning of this year or sometime in 2021 as as well so they have been kind of limiting all these features uh motivated by these research and how they have been misused in many ways going back to your question like what is the middle ground here so I think I think the main point over here is the misuse of these features or these aspects we probably want checks and moderation in place even like when these ads are created and when the ads are like targeted to an audience like for example can the ad platforms like actually evaluate the particular setting where the ads are being sent out I don't know how scalable is that I don't know if people who are evaluating or even when they are being evolved evaluated there could be biases and perceptions of people that could lead to misalignment like for example I would I might think about this particular AD should not be run in this particular way on the platform whereas someone else might say this is totally fine I mean again like there is so much of disparities and how we perceive and how we think about different things and the other thing is I think for some factors we probably should just get rid of these attributes like for example race based targeting gender-based targeting it might might not always be necessary probably they might more cause harms than benefits if we can kind of get rid of these attributes all together we probably might have like better targeted or personalized content recommendations on these platforms yeah that's a tough one I definitely see the argument for removing some of the demographic targeting yet there are likely I'll see ads that are not going to apply to me often hygiene products are generally to one gender or the other for example so uh another thing that I would like to add that's a that's a great example and that was also in my mind and this is also kind of related to another one of my research where one of the goals of that was to not use demographic data but still make content recommendation or content timing or ad tabing better the way we approach that problem was we decided that we are not going to use demographic data we are not able to use race data we are not going to use gender data and we are not going to use people's long-term data on social media platforms we can be privacy intrusive people may not be comfortable with their years of data being shared so can we use only like one week of people's behavioral data like what kind of things they are purchasing what kind of content they are seeing what how much engagement they have on the platform can these behaviors be used to model content recommendations better or add recommendations better what could be the proxy of gender for showing kind of products which they people may not be interested interested is if someone is looking into that kind of product or looking into that kind of doing that sort of Engagement which can lead to people's interest in certain kinds of products so it's more like interests and less less than on the gender that kind of characterization we are doing and yes I mean obviously it would not be super accurate I mean there are also like gender is not a binary thing so so even that we have to consider and people who might be interested I mean there might be other attributes or other features which can lead to similar recommendations without using gender or demographic attributes I think people are becoming very aware of the ethical considerations around targeting and the large tech companies are hopefully making adjustments but uh there's still this big Boogeyman that uh Facebook hacked the 2016 election I guess first of all before we even approach the question you have to have some data to speak scientifically about something what kind of data sets exist that you can look at right so that's a great question and I think a lot of these could be like anecdotal as you said like or peoples can have like experiences or kind of makeup might have seen something like for example anecdotally I might have seen like this kind of thing and this kind of thing I can still see that but as you said like sometimes we look for empirical evidences or data driven evidences that kind of support our hypotheses or claims that we are making so one of the things that happened was in 2018 the house Democrats released a set of like 3500 ads which were run by the russia-based irda during the 2016 U.S election presidential election and that's what we based our Stadium on like this this set of ads that the house Democrats released is public and one reason that they released this was so that researchers or people can actually investigate what happened with these ads like who were these ads targeted to what were the content of these ads and that way we could actually go in and see like whether the ads were only influencing to Auto one way or the other how much they actually changed people's opinions from being left laning or dry cleaning did these ads actually influence people's ideologies and did it actually change people's ideologies and one of the interesting findings in that was not really I mean like people who were left laning they would probably go even more left-leaning or people who were right-leaning they would move even more Right cleaning I mean that also aligns with the psychology and political science literature I mean like it's very hard to change someone's ideology that was one of the preliminary findings of our paper and that's why we wanted to go in and see like okay so if it's not about ideological influence what was it about okay so let's go in and see what was the content of this ash like who were this targeted to did it actually matter like based on say between group and within group group Essence like ideological groups like if these ads were shown to not just the audience that they were shown to if this ad was was rather shown to a representative audience what would their people's perceptions be like so you'd mention you can measure and look into the degree to which these influence to change in ideology how do you measure that so what we did we had the collection of ads we conducted experimental studies so we recruited participants or individuals representative of the U.S population so basically we had like 40 percent Democratic liberals leaning to 40 conservative living and for 20 moderate and then we kind of showed these ads to these group of participants and we asked them what are your perceptions like would it change your ideology would it keep this your same ideology would you approve this content would you disapprove this content would you report if you are shown this content on the platform so that was the methodology that we adopted in our study it was a large experimental study where we gave the fake example earlier of Hillary Clinton as the devil which I would hope is not a headline that would influence anyone of any perspective to anything because it's just silly were the headlines did they tend to be of the silly variety or were some of them well crafted I think even though I said that was the example baby I maybe I did not use the exact words but there were actually ads like that one of those observations of the study was if this ad was shown to a particular micro targeted audience they would not report it they would be approving of it but I think I think a better example for this would be the case of immigrants there were ads which was about it's time to get rid of the parasites or if the immigrants are causing harm to our economy our jobs as you can see it's like these ads were shown primarily to conservatives whereas there was also ads which were about there's a lot of islamophobia in the country Muslims are being scapegoated about I'll just get xenophobia and these ads were only shown to liberals I mean and also I'm not saying like I'm not going into what is the truth and what is false in this ads right now but I'm saying like when these ads were crafted this way and only shown to an audience who is more likely to approve it there was and even though there might be some instigating content in it even then these ads were not likely to be reported by the micro targeted audience I was wondering if we could expand on that reporting mechanism I have to confess I've never or I can't think of instances where I've ever reported anything myself on social media maybe it's because I'm not a power user uh is that a good mechanism is that a way uh people can flag things and the platforms appreciate that feedback and take things down I mean as we can imagine all these platforms have billions of content every second yeah and content moderation is there are automated algorithms but algorithms are not always able to capture and these kind of uh different anti-social behaviors and platforms keep on evolving all the time algorithms can only do what people are making them learn to do it can never capture all the kinds of inaccurate or inept sorry inappropriate anti-social content on these platforms so what these platforms rely on is there is an option to report and report a Content some platforms call it report abuse Twitter does that Instagram does this so all these platforms have ways we as users or individuals can go in and manually report some content and I think they have some algorithms like if 10 people 20 people or people from different parts of the world people from different geographies are are multiple are reporting a Content multiple times that should be just removed from the platform so that is one way how these platforms are relying on crowd contributed data to remove a Content or moderate a Content there might be micro targeted content which is ideally would not would have been reported on the platform if it got a more representative audience they would like likely not be reported because people are more approving of it people are liking that content and probably believing it more and more likely to believe it so so that is one reason that might may not be reported yeah so something like an anti-immigration policy which we think of as having Infinity with conservatives and an anti-affinity with liberals maybe if the Liberals had seen it they would have been flagging that but yes as it stands it only reached the choir and therefore the reporting mechanism seems like uh they've went under the radar what then uh when you look into some of the detailed examples from the survey work what do we learn about the influence that uh preaching to the choir has on the choir so the major three research questions that we had first research question was about how divisive was the content of these ads okay so it's a very simple way to see if this ad was shown to same group versus if these ads were shown to different groups like for example liberalism conservatives both are people's opinions divisive and that was one of the findings that kind of motivated that was like wow this is so divisive content as like if an ad is likely to be approved by one group it's more likely to be disapproved by the other group within a group there is like significant alignment like people within that group were more likely to oh wow this is something that we approve of or we all together disapprove of so I think that was one of the finding about this ads like these ads were so politically aligned or politically connected or situated that they were leading to divisive opinions among the different political groups so that was one of the findings the other finding was that we had three different measures that we are studying so one was about would someone be likely to report it would someone be approving it so I think approving is a little different from uh reporting it's like even if I may not report something I may not approve it it's like a different layer of whether I like a Content or not like a Content that could should be wanted I wouldn't report the immigration one because even though I don't agree with that point of view I think it is a political point of view that I can't tell you not to have exactly yeah so that's a very good example and the other is false claims like how much of false claims these ads had like a particular ad has so that way like three dimensions that we were studying we had so contrasting figures I mean like even when we made it we were like so surprised that the CM ad if it is liked by or if it is approved by the Liberals it's super disapproved by the conservatives or the vice versa or the reporting kind of phenomenon is exactly the opposite false claims again like people if someone's from Liberals are likely to notice a false claim the conservatives would not probably notice it and vice versa so that were like like these kind of divisiveness of this content was one of the major findings of these ads we also conducted the other experiment like if this ad was shown to representative audience what would happen in that case when we compared the responses of the ads between targeted audience versus a representative U.S population we saw that the ads would have say more likely to be reported less likely to be approved and more likely to be found false claims in that and that those were in other kinds of findings like how the ads were influential in the audience so would it be correct to say the internet research agency was optimizing their content for divisiveness yes so that was one of the primary objectives of the ads that we could kind of think of I mean like when we kind of replicated how these ads were targeted like they were targeted to people's interests in say African-American civil rights or people's interests in if they're particular like liking or particular Pages which are probably it's a conservative learning or people who are more liberal leaning I mean that kind of pages so that was one way to identify people's political alignment and ideologies like where they lie in the political spectrum and based on that they targeted this audience when we've calculated the click-through rates of these apps the it was super super high compared like over a magnitude higher than like ads generally on the platform I think generally on the platform there is a click-through rate of like point nine percent or something but these ads had something like over 10 percent and like by click-through rate is like if someone is shown an ad How likely they are likely to click on it and these ads the the well-targetedness of these ads also was reflected in the high click-through rate I mean that's like a significant number yeah that makes sense you uh bring content teed up for the Right audience they're inclined to click on it what would they find after that click where did what was the destination URL for some of these uh some of these had destination URLs of these uh but I would also say like basically what I mean the clicking is on Facebook home page you'll probably see your content and if you click on it this ad only just uh gets Let's uh the same content just gets in increased and then you could rather click on some URL if the ad has and then that takes two takes you to another page but that measure I do not have yeah but still a strong engagement signal very much very much yeah so when you look through the results you've got and the data and the survey that augments that can you ballpark it how influential was the attempt that the IRA made to affect the U.S election in 2016. one of the things that the IRA did was the targetedness of these ads and influential I mean we cannot really say like did it actually change people's ideologies I think probably not but maybe it reinforced people's existing ideologies that is one of the strong findings like for example someone who may not have footage or may not have been that interested in the election diet definitely got their attention and interest about the election and that might have influenced the voting in some way or the other so would it should we characterize that effort as a success or were they really chasing their own tail or something in between depends on what they wanted to do I mean what their goal was and I don't know if that is public I mean some of this could be assumption but I think I think yes I mean if there's a goal was to Target the ads or cause divisiveness in the like show more and more divisive content to well targeted audience but still stay on with the content and not be flagged or reported on the content they definitely succeeded in that why have to believe that uh there's more than one person at Facebook who's read your paper and probably all the references and is taking this seriously uh and it's been a bit of time since all that went down have there been any significant changes to the platform to address some of these concerns yeah so I think not just this research but they were also research about how there are raised based discrimination for example uh like different housing localities are probably not shown to people of certain race or religion I think that has also caught attention at Facebook and and also like this kind of content this kind of studies so one thing that they have actually done is they have limited all these attributes about what could be used and what could be not I mean like the attributes are much more limited now compared to how it was back in 2015 2016 when you had like all flutter of attributes like demographics gender not exactly with zip code but people's locality it could go to an unless that locality had less than 15 or 25 people I mean they would also go to that sort of targeting I mean I think that has been much more limited right now one of the major implication of this work was how do we identify as and we were also conversing on this like how do we identify content which can be flagged which can have something that is anti-social about the platform the sort of thing so I think one of the implication of this work was maybe content which is highly marked micro targeted maybe content uses a lot of these parameters to Target a particular thing they are likely to I mean maybe they should be prioritized first to kind of go through the moderation pass because it's hard to know which content would should needs to be moderation which kind of does not miss moderation but maybe this might be a bit you know parameter to rank content to do a moderation pass and I think that was one of the key takeaway of the study and there may be I don't know if the platforms have actually adopted this but that would have been that would be a really cool way to actually go through these platforms uh go through this content sorry well in addition to this paper I've seen in your CV there's quite a bit of other things in the ad space related to our season here so I was hoping we could dip into some of your other projects too one of the studies that we did in our early research was can we use ad targeting tools to understand people's likes and interests and awareness so one thing that we studied was how the kids awareness of schizophrenia varies across different parts of the United States about for across different groups and communities basically it would say like these people are interested in schizophrenia or mental health related interests from different parts of the state and that kind of provides information or empirical information about where we could do more mental health rise where we could run mental health campaigns so that people are more aware and kind of are provided better support about mental health because mentalism is the stigma has a lot of stigma associated with it so that could be limited through that kind of study so that was one study and another very different study was when I was interning at snap research we were studying can we only use limited amount of data and and non-demographic data to understand people's preferences of like when there would be more likely to see an ad even though probably nobody none of us like ads uh we cannot just think about it's not as simple to just get rid of ads on the platforms ads are a way that leads to Equitable content on the internet if the internet content becomes pay for Pay Per use then people would only be seeing content which they can afford so I think ads enable platforms to sustain ads enable free content exposure for many of us so for for all of us on the internet I would say so one thing that we were studying in this particular paper at Snapchat was how to better time ads so basically all of us have different preferences and different use on this platform such as Snapchat if someone might be using it in the night someone might be using it while they're doing uh working on a treadmill and sort of thing if an ad comes we probably might be irritated and kind of get rid of kind of keep aside the phone and get back the get back to the work so that does not really help the advertiser or the platform and probably the platform would again run the ad once again at some point other point of time so how we can improve this engagement so one thing that we were studying we did a causal study we used one week or two weeks of data of individuals and based on when they are more likely to be receptive to ads if the ads are shown at that point of time we saw like significant Improvement of at CTR and add receptivity of these individuals so that was one e takeaway and we also did an experimental State simulation and if ads are run in better timed manner that could lead to four percent increase in Revenue uh of the company and if that happens I mean that could lead to a better world where we could do with lesser amount of ads on the platform for same amount of money and that would that is kind of win-win for the user for the platform for the advertisers for all of them so should I as an Advertiser take that wisdom and try and I don't know a date do day parting on my campaigns or is this something the platform should take and do for me it could go both way like I think it's more the purpose of the research is more to inform that people have different preferences about when they are likely to see an ad and when they're not likely to see an ad like for example if if an ad comes when I'm kind of lazy in the night I'm just surfing through my phone I see a Content I see an ad I'll just wait for the ad and go back to go to my next content so that's sort of the motivated kind of key finding for the study I think it could go for both advertisers as well as platforms if platforms have better algorithms to time ads then advertisers would probably say like okay if I am investing this amount of money getting these many views I should be fine if advertisers can also encourage platforms to use these kind of AD timing based algorithmic allocation of hairs so I think the advertiser should also like that so other thing is like we were not looking into particularly content like what kind of contents of ads people who prefer at what times of the day so but maybe that that could be one follow-up study that advertisers may also be interested in like maybe this kind of ad this kind of content of art is more suitable in the summer time or more suitable in that during the daytime and not suitable during the night time and I think that would be another level of personalization and probably better at allocation that people could study and people could use oh cool stuff if listeners want to learn more about that or some of your other work can you tell us some good places to go online I have my personal website it's my firstname.comustuv.com and a lot of This research is put up off late people have been Super Active on Twitter to share their research and everything so I think that is another place you can electric folks engage with research I think personally for me I'm at MSR right now so Microsoft research I'm specifically working on this fairness accountable transparency and ethics of AI and our lab has a page and MSR feed if someone searches it on the internet keeps on publishing all the research and we have all our recent works on there well good we'll have Microsoft research fate in your personal website and some other goodies in the show notes for listeners who want to follow up thank you so much for taking the time to come on and share your work thank you so much Kyle have a great day that concludes this installment of data skeptic ad Tech here's a quick preview of what we've got coming up next episode our crowdfunded games different from traditionally published games are they more Innovative or new or novel that's next time on data skeptic ad Tech [Music]