Ad Network Tomography

Key Takeaways

if you're not already on the data skeptic newsletter and we send it out same day as the show so if you haven't seen that you're not on the list head over to dataseptic.com scroll down just a little bit and put your email in the box you'll get an extended bio on the guest and a little behind the scenes stuff in fact in a couple of weeks I'm going to reveal what the next season's topic will be to the newsletter list first so if you'd like to know in advance of January 2023 sign up now at dataseptic.com we learned last episode the manner in which advertisers track you online is evolving ironically the movement away from third-party cookies means that that third-party tracking which used to be done somewhat out in the open has now been moved server side where we don't see it corporations rarely disclose details about the algorithmic mechanisms that Define their systems so any insight about data sharing that takes place between these sites has to be measured through experimentation how much data sharing is going on find out in today's interview [Music] student at the University of Iowa and I'm working on auditing data sharing ecosystems online which essentially means I want to make the ecosystem more transparent in the sense that we know who is collecting our data who is sharing it what's happening to our data and more recently I've also been involved in the compliance side of this process as to how compliant these entities are or what the state of the art regulations are surrounding these compliance regulations and I'm not here assistant professor at the University of Iowa's computer science department I do internet measurement with the focus on trying to understand how data that is gathered by entities actually impacts our experiences on the internet well how did you first get interested in the topic of data sharing data sharing is the second part of what my interests are and I'll elaborate that further so I was interested in like I said auditing data sharing ecosystems or data ecosystems online and there are two major parts of it in my opinion one is data Gathering and the other is data sharing so there has been a lot of work that has been done on data Gathering and one example of that maybe you've heard of it it's cookie syncing anything that enables entities to gather your data has been studied a lot and has been there has been significant work on that but what was not clear or what was not studied in detail was what happens to that data after it has been gathered so we essentially wanted to answer that question and our specific work atom which focuses on data sharing was also motivated by a previous work similar to this which used advertising bid values to identify these data sharings on the server side and atom was a more more generalizable way for doing the same thing well if I visit a website let's say like an e-commerce store and I look at you know all men's clothing or something like that if I come back later and it seems to remember that I'm interested in men's clothing I don't you know it's the same exact site I know there's a cookie there I personally don't feel like that's a privacy concern but it would be a little surprising to me if then I went to a different clothing store that I'd never been to before and they knew already in advance I was interested in men's clothing is that a possibility that's not just a possibility that is actually what is happening and the only two Works surrounding data sharing one of them focuses on exactly this thing which is called ad retargeting so they sort of figure out how data sharing happens based on the exact phenomena that you mentioned which is you visit a website you show an interest somewhere and then you visit another web site and they seem to know your interest over there so there presumably happened some sharing on the back end which enabled the other side to know that or or The Advertiser on the other side to know that you are interested so that not just is a possibility that is what's happening in reality as well are there any legal boundaries about what is and is not allowed in terms of the sharing this is a more recent thing than than anything else so if there weren't any data regulations in place before the most famous one that you will probably be aware of which is the gdpr but that was only and is only restricted to Europe and the most famous One or the first one in U.S was the CCPA which is now being followed up by each State's own data rights policy but they only are in their initial stages and they have multiple more iterations to go before they have much better regulations in them basically they there's nothing illegal about what is happening online right now online tracking retargeting Gathering is not really illegal the CCPA only goes as far as saying that if your data has been gathered by certain entities and this is again specific to California residents if their data has been gathered by specific entities these entities just have to follow these rules and provide the ability for these California residents to opt out of this data sharing or do opt out of the data Gathering or request that their data be deleted that's as far as regulations go and this is true for Europe as well with the GDP here could you give me a sense maybe it's your own Intuition or there's something empirical about this but to what degree is that process uh frictionless if I say I don't want to be tracked do I have to fill out a postcard mail it to the right department or is it really something as simple for the average person to execute on with respect to CCPA or even gdpr they are very less standardized methodologies out there that allow a person to you know globally remove people or um or stop people from selling their data or sharing their data in CCP at least or in us at least they are or in CCP there are very less to zero standardizations and there have been recent researchers on gdpr as well on how they can be standardized as well for example there's a right of making your data more portable but there isn't a portal that allows or there isn't a standard that allows your data to be exported or imported in a certain Manner and same goes for other rights as well if for example there's a right to delete your data but there is not a standard anodized platform that where you can go and just take off all the websites that that have your data and get them to delete every Advertiser every data broker has their own web form or own way which is usually an email or a web form or a telephone number at the least which you can you know call or email to exercise you're right yeah so if you look at the CCPA right now there's uh about 450 data Brokers that are registered with the attorney general in California because they handle I believe it's over 50 000 residents data California residents data sets to opt out entirely from this ecosystem of data sharing an individual would have to go over the CCPA relevant parts of the Privacy policies of 450 different data brokers figure out what methods or you know what mechanisms each of these policies facilitate for exercising their rights and then act on 450 individual methods when we think of offline versus online behaviors I I don't know that we always can make an apples to oranges comparison of uh you know that the Privacy abstract concept carries across but I in the real world no one would think that their credit card company would share the statement with your employer that's just such a big No-No but when I have data behind the scenes if I'm fingerprinted and there's server-side work going on I really don't have any Clarity into who might be connecting data sets behind the scenes do I have cause for concern about that sort of thing I I believe so and we shared the same concern which is why we did this work to uncover or to sort of make these these exact shadings more transparent so this was just the first step towards this concern that what is actually happening once we gave our data to this credit card company who is it sharing it with what are the consequences of those sharing and Etc so these are all private even if they're publicly traded companies they're making private decisions I don't imagine they necessarily publicize all of their data sharing policies and whatnot aren't these just black boxes how can you get a peek inside so essentially what you have is uh these entities behaving as a black box right so we did we do know that the data goes to them but we don't know how they shared with other entities however if there's anything we know about the advertising ecosystem is that or the internet in general it's that the internet is basically monetized through advertising and all of this data is actually being gathered for the purpose of advertising so if we can study how data that is being leaked through a specific entity ends up impacting the advertisements that you see that are being targeted at you we can actually identify the relationships that exist between the entity that you leaked data to and The Advertiser that put the ad in front of you so in that way by doing this systematically we can figure out these relationships that might exist even though all we can see is a black box from the outside and since I think we'll end up talking about this phrase a lot could you give a roughly formal definition of what is behavioral targeting thanks to our sponsor neptune.ai 99 of machine learning teams are doing awesome things at a reasonable scale with four people and two production models but most of the industry best practices come from a handful of companies operating models at hyperscale the folks over at neptune.ai want to change that by sharing insights tool stacks and real life stories from practitioners doing machine learning and ml Ops at a reasonable scale they even built a flexible tool for experiment tracking and model registry that will fit with your workflow at no scale reasonable scale and Beyond check them out at neptune.ai foreign what is behavioral targeting so behavioral targeting is one type of way in which advertisers present an ad to the user and what's special about this is that it's based on the user that is seeing the ad so it's based on the interests of the person who is being who the ad is being delivered to and how does the platform that's serving the ad know what those interests are so that's the first part of the data ecosystem which is gathering the data so a Tracker or an Advertiser will be present on a lot of the websites and using techniques like cookies and tracking pixels and sharings on the back end they gather information about a specific user and when they see that user again somewhere they know what their interests are specifically their browsing history or likes or dislikes are and they use that knowledge to present you the user that ad and that's basically behavioral advertising and there's some sort of I guess I'll call it a non-obvious process also a non-trivial process that's going on here like a list of URLs you visited does not obviously convert into like a vector of Interest or something like that are there standard techniques or is this sort of a proprietary approach a way so this is a technique that has been used previously by other people as well and on by identity by identifying a user's browsing history you can make a well-educated guess on what kind of things they are interested in so this is a technique that has been used in the in the community previously as well and it's it's a well accepted technique that browsing history can be used to represent and interests of a user it doesn't have to identify who you are or your exact details but your interests are at least extracted from your browsing history and that can be enough to you know present a more related ad to you yeah I mean on the positive side this sounds like a good thing like I personally have no interest in soccer so sending me any soccer ads is a waste of uh the advertising dollar and an annoyance to me but if you look at my browser history you would quickly see that I'm interested in machine learning and software development stuff like that and you know even an ad that I'm not going to click on I'd rather have a developer tool than a soccer ad uh where does this become Insidious if it does so that's a that's a very good question we were talking about this recently as well and in my personal opinion it's not bad that trackers have your data or you're getting a personalized ads what the issue is is transparency so our goal as well as described in the paper as well is to make these data flows more transparent so we should know who is giving you that football ad and where did they get their data to because for example if something Insidious does have happen or something that sort of some some of your personal data is being leaked or is being misused you know who to hold accountable or and in addition to that the the policy the data right data right policies they also need these Frameworks to hold data Brokers or trackers accountable if something uh Insidious does happen so having these data flows or having these retargeted ads is not a bad thing until it becomes a bad thing and then you need these auditing Frameworks to sort of hold people accountable so the main issue is accountability and transparency not them existing in in the ecosystem yeah just to present an alternate Viewpoint about advertising not being bad you know we just have to really think about how the internet is kept free today we wouldn't have all these incredible Services if it weren't for advertising generating enough Revenue to fund all of these lovely internet services that we use right so it has done something good I think where the issue lies is that people don't know what they're actually giving up to get something in return and they don't have so there's no transparency and there's no control and really what we're trying to develop here is a set of mechanisms that facilitates both transparency and then the users right there control how their data is used and how it floats well that's a great segue right into Adam can you guys give me a definition of what the Project's all about yeah so I think we've we've gone over the sentences somewhere in the previous question as well but I'll go over it again atom focuses on analyzing what happens to your data once it is gathered and focuses on uncovering these uh data sharing relationships that are inside the Black Box you mentioned earlier to make the system the deduction ecosystem more transparent hey can we talk a bit about the mechanics what's it doing to accomplish all that okay so there were two major parts of atom one was first where we had to establish that your ads can actually represent you and the second was inferring these server-side relationships so the first part we instrumented an open source tool called open wpm and we created these online profiles each represented an interest for example Health Soccer Games Etc and then we collected ads related that were present it to them and then we compared Those ads and we saw that an interest group within itself saw similar ads but when compared to other interest groups it saw different ads so essentially your ads and my ads would be different and we were able to quantify that and using this Insight in the second part of the experiment we exposed ourselves or leaked our data to certain combination of trackers and then we saw how that affected our ads and if that is affecting our ads then we can say something about a relationship between that tracker that we exposed ourselves to and The Advertiser that gave us our ad and is this a process where you and your team are you know visiting specific sites and checking cookies or is this a automated system how does it all run it is an automated system the first part and the second part both automated in the first part we visit this uh the given site so we had 50 sites for each interest group so Health had 50 sites games had sites we visited those 50 sites we gathered all the cookies and things related to it and then we went and collected ad from other sites as well and the second part was leaking our data to certain trackers and then collecting ads for um for those profiles gotcha so those initial profile of URLs that's kind of like a burn in process where you're sort of teaching the internet I guess what your interests are yeah in terms of collecting the ads were these text ads or display ads what did you get back key contribution of atom is that we are agnostic of the ad delivery ad delivery methodology as well so we collect we to our to our best sort of instrumentation we collect all kinds of ads that involves header bidding ads and rtb ads that's real-time bidding ads and header bidding ads so we we try to collect all kinds of ads they're typically images there they're typically images yes and we extract features from those images uh for our analysis part as well and those features include texts and other features like labels and entities Etc so we that those are image ads and we extract features from those yes is that a tricky process I know image recognition is pretty good but just saying that oh there's a table and a chair in this photo doesn't necessarily mean it's home decor for sure for sure that's a difficult process we did some validation with the annotations that we extract from the images we used Google Cloud platforms Vision API to annotate our images and after annotating it after annotating it extracting all kinds of things like text label entities context and things like that I went over and I manually validated the labeling as well and turns out that things like the text extracted from the image or entities and things like that were more accurately extracted from the images and were also more representative of the ads as well as compared to other things like faces or you know this the description of the background or anything like that so at the end eventually it depended on things like text and entities from the pictures or ads that we extracted if you saw an ad for a laptop yeah where the laptop is open and uh you know floating in the air right and you've got the IBM logo right next to it or the ThinkPad logo right next to it machine learning you know is a lot better at you know extracting the presence of this logo and the text uh the laptop floating in the air with this color in the back right it turns out that the truly really how ful way to as you might imagine these ads that contain the word ThinkPad in them are more likely to be targeted towards uh individuals that are interested in Tech perhaps that's true but we all have laptops so there's going to be some background rate I presume if there weren't any server-side data sharing going on we'd see you know a uniform distribution here I expect across the groups um I imagine that's not what you get though how different from uniform are things when the interests are taken into account so to measure how different we would be from a control profile by that I mean they didn't have any browsing history and we just collected ads from them so we wanted to see what a new user would see and turns out that our interest groups across all intascript see a higher high difference from the control so we all have laptops but it turns out that a control profile is not receiving as many laptop as as it as a tech profile would so I I think there's a clarification to make right so so far we've been talking about the first part uh of this project which is the correlation between the ads that you see and the websites you've been browsing right so the second part where we actually try to uncover these data sharing relationships the way we do it is let's say we're trying to figure out the data sharing relationship that exists between alphabet and all of these other uh advertising organizations that exist right the way we go about doing this is we create a browsing profile and we love alphabet to see this browsing history right so we enable all of its trackers we don't block any of them so alphabet gets to see all of the browsing history and the advertisers get to put ads in front of us what we do next is we create the identical browsing profile this time we make sure to block all of alphabets trackers of course there's no guarantees of completeness but we make a best effort uh from all of the crowdsource data that exists to block access to all of alphabet's trackrooms and then we study how the ads delivered by these different entities actually changes right and then we compare statistically the ads we got from the first experiment where alphabet get got to see all of your data being or got to gather all of your data and the second experiment where alphabet did not get to observe you at all and has no data about you right and it turns out if there is a statistically significant difference between these two it must be because alphabet was sharing uh data with these advertisers what's the pool of advertisers look like we'd mentioned uh alphabet who else is in there are these names people would recognize the relationship that we uncover that includes includes advertisers like alphabet they'll also include applies like gum gum Oracle Facebook and open X so these are these are advertisers so advertisers and those are The Trackers so they're they're on the advertiser side as well awesome because The Trackers we predetermined that the 10 big trackers that we wanted to in for relationships with and the advertisers that we saw them having relationships with include openx trade desk uh PubMatic Amazon and uh other big names like critio and so forth so this is the big big names that come from the advertisers that we uncovered let's get into the data uh do we find a certain Independence across all these or are you detecting some data sharing we can see that alphabet that's a good point that we see that alphabet even though it's a one of the biggest if not the biggest data broker or company that consumes data has relationships with essentially it it has the most relationships that we uncover it has around I think six relationships that we we can cover so it it looks like that even though these big entities are independent they are interacting with each other as well and they are sharing data with each other because it only increases the data points that they have on a user and the more data points you have on a user the better you can Target them and the better their ad money is spent there's always gain for them to share data and another thing to keep in mind is that when we talk about data sharing it's not necessarily looking like alphabets you know uh zipping up these uh giant data sets and shipping it over to somebody else right it can also be as simple as alphabet providing a platform through which people or advertisers can Target different individuals so alphabet doesn't have to take your data and give it to someone else it just provides a platform through which all of these advertisers can select the users they're interested in an alphabet does the targeting for them or on their behalf was there anything surprising in the relationships you inferred not as much and we were able to Value so we were able to validate most of our relationships through uh multiple multiple ways um except for a few of them we used like we used client-side cookware syncing or ccpas uh or ccps disclosures to valid validate our relationships but generally we saw what we were expecting um in terms of relationships actually uncovered I guess one thing that's a little bit surprising to me is that if you look at the paper right uh We've identified uh 10 different uh or 12 different sharing relationships that happen of phase 12 only one of these could actually be validated by the CCPA and that to me speaks about the CCPA actually not going far enough or not being followed through correctly right it's got to be one of the two the CCPA is not mandating disclosures of all data sharing relationships and so we're able to capture some that happened that we can't validate through the CCPA disclosures and there exists 11 others of which nine could be validated through other very rudimentary techniques well if the CCPA is too weak or incomplete in some way I mean my feeling is that you go and you fix the the policy right you improve it you make it more specific or whatever if the policy is sufficient but isn't being followed that's another matter and I know you may not have a sense of which bucket we're in but are these people violating the CCPA or could we put a percentage likelihood they're violating it on them I think that's a hard question to answer uh just because we lack complete information right but you know this is these are the set of questions that Adam is actually trying to get at right so the goal is to put out all of these relationships these 12 relationships for example that we identified out in the open and then perhaps push these to the attorney general in California and you know get them to investigate this more they are going to have more access to information than we do they're going to be able to get the information that they need to verify that these are still in compliance or not in compliance with a CCP or or even verify that they're not required to disclose these relationships by the CCPA and I don't know how exactly we'd frame it or what the units are but do you have any sense from the results in your work of the the degree to which the sharing is taking place is this something we would say as a rampant or is in an r d phase uh I guess how significant is it based on the results that we have I would say sharing sharing definitely is is prevalent to reduce sort of uh false positives we were very stringent on we were very strict with the threshold that we set out and still we got like 12 of these relationships we used very basic features which was text of the image if we have more context of the ads that we're receiving or if we have more powerful models we can we can essentially uncover more because there definitely is more that is happening we were very conservative so we get true positives in our inferences I believe the other thing to look at is when you look at our results right the advertisers that end up receiving gathered data or you know being part of the sharing ecosystem are giant ones but we're talking critio we're talking about Amazon PubMatic trade desk these are huge and The Trackers that we're seeing frequently sharing information with them belong to Oracle alphabet gum gum openx and Facebook again massive trackers to answer your question I would say it's probably really really prevalent the sharing of this data just given the size of the tracking entities and the advertising entities that we're seeing these relationships between so in this work you guys had to go to a great effort to collect this data set even we've only touched on that I get the sense of some of the engineering challenges we might not have even touched on this was not an easy project and it's there to then be an audit uh I I guess another way we could look at it if the Attorney General thought this was relevant or if the CCPA was expanded is maybe these companies could be forced to do some form of disclosure maybe not giving you Rude access to their systems but sharing some level of information do you have any thoughts on what would be useful for them to share for an outside auditor to be monitoring these sorts of data sharing policies for starter they if they disclose the third parties that they shared their shared the data with that would be a positive thing that would help in auditing as well and generally just share what they're doing with the data which covers how they're collecting it what they're collecting and who they're sharing it with I think these are important things for consumers to know as well and for auditing Frameworks or for policy enforcers as well they they would be a good good start I would say yeah one thing to keep in mind is you know it's likely that behind each of these sharing relationships is a business contract this is something that's non-technical evidence potentially for the existence of sharing relationships right usually these contracts are going to specify exactly the type of things that might be traded between these entities and so it seems to me like these are things that you know Regulators behind is the CCP are similar uh protection mechanisms would be able to get a lot you know access to and uh use more effectively and let's see you'd mentioned that the atom system is pretty much fully automated that means at least from a technology point point of view it can scale up do you have any plans for where the Project's going in the future the follow-up of this work is focused on the validation part and we're trying to look at how did the data Brokers are compliant with regulation regulatory policies like CCPA you know one of the things I'll actually push back on what you just said Is You and you were right that Adam is fully automated but I would argue that we actually did not have the ability to scale very much at all this was a massive engineering effort that took a ridiculous amount of compute and honestly I don't know how Mars managed to pull it off it was pretty ridiculous the system that he ended up building uh to do this it was incredible however it for example in this paper all we were able to do was study tank trackers and identified the relationships that they had for the advertisers right so that's not really speaking to scale in upcoming work we're trying to make this a little bit more scalable one of the goals that we have is to establish relationships with enforcement agencies so that we can actually push these sharing relationships that we're identifying they're not disclosed as part of the CCPA do the regulatory authorities so they can start their investigations or they can use the statistical evidence to motivate an investigation going forward right so that's one angle that we're looking at then the second angle that we're looking at is uh just trying to understand exactly what the CCPA requires a data broker to do and so we're working with some of our legal friends to help us understand the regulations and then actually studying whether these data Brokers are complying with each aspect of the CCPA and I know you can't give me a solid answer to this question because how could you know but based on the initial results you have here could we label the degree to which companies are in violation are these flagrant violations or are these you know the Growing Pains of rolling out a policy uh to what extent are things not as they might seem I guess we we're still in the in the initial stages of these follow-up projects but the analysis I've done so far which is very limited I can see that there there might be several shortcomings and for these data Brokers and organizations that deal with data even though they're the actual registered data Brokers on the Attorney General's website you can very obviously see some very straightforward shortcomings but we still have those results are just preliminary and still need more more better or deeper analysis to make a actual statement there's a lot of meeting research there's so many open problems and uh yeah there's a lot of compliance issues to study what's the future for both Adam and your own work the future for item again is firstly establishing a good way to scale this up or to to use this for as an application for for um you know maybe for a terangeline's office something and then and the other part would be seeing or seeing the compliance of these these relationships that we have uncovered I guess yeah and is there anywhere listeners can follow both of you online uh yeah so I'm active on Twitter and email and Linkedin so I can reached out on all these three platforms I'm on Twitter at rishab and underscore there's another Russia been I'm not that guy all right well we'll have correct links to both in the show notes yeah thank you both so much for taking the time to come on and share your work yeah thank you for having us [Music] well that's it for today's episode you know the Advent of this text to image technology we're now seeing everywhere on the internet without a doubt that's gonna make an impact on the advertising space but prior to these computer vision breakthroughs we had breakthroughs in natural language processing and models like gpt3 that could be used to write things like job ads they essentially showed How likely it is that different genders would apply for jobs if they redrop advertisements with these stereotypical words one proximal goal that's to maybe have equal probabilities of applying for a particular position that's next week on data skeptic ad Tech [Music]