Machine Learning and Fairness

Key Takeaways

all right hey everyone and welcome to machine learning and fairness so i'm hannah and jen is here in the audience she's going to come up in a couple of minutes and we are both machine learning researchers in microsoft research new york city and since most of you don't know us i wanted to take a minute or so just to tell you a little bit about who we are and what we do okay so as i said we're machine learning researchers and we've each been doing machine learning for about 15 years at this point working on a mix of theoretical and applied projects often in collaboration with social scientists over the past four years though we've both become increasingly interested in fairness accountability transparency and ethics as they relate to ai and machine learning we're both members of microsoft's fate research group which was formed in 2015 to focus on exactly these issues it's an inherently socio-technical group with core ai and machine learning researchers like jen and myself and researchers in the social sciences policy and the law jen and i are also both very involved in microsoft ai and ethics in engineering and research committee or ether where i co-chair the ether working group on bias and fairness and jen co-chairs the working group on intelligibility okay so i want to start by talking about ai and machine learning so as you all know machine learning is super hot right now and over the past 15 years or so we've seen machine learning go from some weirdo academic discipline only of interest to machine learning nerds like me to something that's so mainstream that it's mentioned on billboards and even appears in web comics and tv shows if you don't believe me this graph shows the number of registrations for the largest machine learning conference neural information processing systems or eurips for short from 2002 with around 600 registrations to 2018 with eight and a half thousand registrations that sold out in 11 minutes and 38 seconds yes really so at the risk of stating the obvious we're living in the age of ai machine learning is everywhere and at least for now it looks like it's here to stay and this is great it means there are some amazing opportunities for researchers like me and jen for people with practical machine learning skills and for companies like microsoft and indeed to put microsoft's mission statement into practice we're definitely going to need ai and machine learning but at the same time as all this excitement we're seeing that these new opportunities also raise new challenges challenges that have received a lot of attention in the media and have really highlighted how important it is to get ai right to make sure that ai does not discriminate or further disadvantage already disadvantaged groups many of these stories have focused on high-stakes decisions where machine learning systems are used to allocate opportunities resources or information in ways that can have significant negative impacts on people's lives so for example a couple of months ago amazon revealed that it had abandoned an automated hiring system after finding that the system amplified gender bias in the tech industry or as another example i'm sure many of you heard about the pro-publica investigation a couple of years ago which showed that compass a widely used recidivism prediction tool was according to some metrics racially biased these biases can also affect more mundane systems too for instance in 2013 latanya sweeney showed that online ads suggesting that people had been arrested appeared more often for distinctly black sounding names than for other names thereby reinforcing negative stereotypes of black criminality so how do these stories relate to fairness well broadly speaking they're all examples of machine learning systems causing harms to some people but not to others so in other words examples of machine learning systems behaving unfairly of course and this will be a recurring theme throughout this talk fairness is a fundamentally societal concept that's been studied by philosophers lawyers and social scientists for hundreds of years moreover there's no one-size-fits-all definition of fairness but suitable for all systems in all contexts so it's a bit daunting to be suddenly grappling with fairness throughout the tech industry that said at least as we see it we're not completely unprepared in particular we can think about fairness in a similar way to the ways that we think about security and privacy first security and privacy are both socio-technical challenges as is fairness in the context of machine learning sure fairness is a little closer to the socio end of the spectrum but they're all on there second in the tech industry security and privacy are much older topics than fairness so we can really learn a lot from their history in particular in the late 90s and early 2000s security and privacy efforts were very competitive adversarial and closed door and this was driven by companies fears of public criticism negative press and financial costs and by the assumption that it would be possible to eliminate security and privacy incidents as a result the focus was very much on prevention and detection often post-deployment during the 2000s though this landscape changed considerably and in fact microsoft was a leader in this change companies began to realize that there was value in collaboration cooperation and openness and began working together to prevent and detect vulnerabilities now we've reached a point where the tech industry acknowledges that it's impossible to eliminate security and privacy incidents of course this doesn't mean that it's not worth investing resources into prevention and detection but we must be prepared for such incidents to occur and must therefore also prioritize and invest resources into responding appropriately i think that we can learn from this and approach fairness with the same emphasis on collaboration cooperation and openness of course we need to prevent our systems from exhibiting biases as best we can but we also need to respond appropriately when they inevitably do to put it differently we need to adopt a growth mindset toward fairness just the security and privacy aren't one-shot processes and require continual evaluation and improvement fairness isn't a one-shot process either we will make mistakes let's therefore anticipate them as best we can but when they do occur let's turn them into something positive specifically let's learn from them and use them to improve our systems and contribute to our successes okay so with that background out of the way we're going to cover four topics today that will collectively give you an overview of fairness and machine learning we'll start by talking about some of the main types of harm that can occur so that you'll know what to look out for in your own systems then we'll talk a bit about who might be affected by these harms so in other words which groups of people or subpopulations might be most at risk next we'll run through where these harms come from focusing on the different stages of the machine learning pipeline and at each stage of the pipeline we'll talk about best practices and strategies that you can use to mitigate the harms that can arise at that stage and then finally we'll talk about some recently developed software tools before concluding with a handful of open questions and key takeaways okay types of harm so i want to note that the next few slides are based on a framework for characterizing types of harm by aaron shapiro solomon baroccus kate crawford and myself though i've modified the framework a little bit here broadly speaking there are five main types of harm that can occur in a machine learning system allocation quality of service stereotyping denigration and overall under representation to explain each of these five types i'm going to run through some illustrative examples starting with the most well-known type allocation but before i do that i want to note that these types are not mutually exclusive it's possible for a single system to exhibit more than one type of harm and indeed many of the examples that i'll show you over the next few slides could potentially illustrate one or more of the other types as well so first allocation as i said earlier many of the recent stories in the media have focused on high-stakes decisions where machine learning systems are used to allocate opportunities resources or information in ways that can have significant negative impacts on people's lives so for example amazon abandoned its automated hiring system after finding that the system amplified gender bias by withholding employment opportunities from women in the tech industry similarly the racial bias found in the compass recidivism prediction tool is also a harm of allocation because among defendants who ultimately did not reoffend black defendants were much more likely to be classified as high risk than white defendants and thus denied the opportunity of being released on bail quality of service is all about whether a system works as well for one person as it does for another even if no opportunities resources or information are extended or withheld so for example last year researchers found that three commercial gender classifiers had higher error rates for images of darker skinned women than for images of lighter skinned men much like accessibility issues harms relating to quality of service also raise questions about respect dignity and personhood imagine how a user might feel if a system repeatedly fails to recognize her voice but easily recognizes those of her peers as a second example of a harm related to quality of service virtual reality systems often make women feel sick while men seem to be largely unaffected okay stereotyping one of the most well-known examples of stereotyping and machine learning is the example i mentioned earlier where online ads were more likely to suggest that people with black sounding names had been arrested thereby reinforcing negative stereotypes of black criminality in a similar vein researchers at princeton found that translating he is a nurse and she is a doctor into turkish a gender-neutral language and then back into english yields the stereotypical she is a nurse and he is a doctor and just to be clear this home was not unique to google translate it affected microsoft translator and other machine learning systems too and google actually came up with a really creative way to address this harm just last month denigration is where a machine learning system is itself part of a process that is actively derogatory and offensive so for example a couple of years ago google photos infamously mislabeled an image of a black woman as gorillas this mislabeling is offensive not just because the system made a mistake but because it specifically applied a label that has a long history of being purposefully used to denigrate and to demean people to liken them to animals as another example of denigration microsoft had to shut down the chatbot tay shortly after it launched because it started generating hate speech over and under representation are fairly self-explanatory but as a concrete example researchers at the university of washington found that professions with an equal or higher percentage of men than women for image search results were even more heavily skewed toward images of men than reality so then to recap the five main types of harm that can occur in a machine learning system are allocation quality of service stereotyping denigration and overall under representation before i move on i want to again reiterate that these types are not mutually exclusive a single machine learning system can certainly exhibit more than one type of harm as i've tried to indicate here on this slide all right great so we've covered different types of harm next i'm going to talk briefly about who might be affected by these harms so in other words which groups of people or subpopulations might be most at risk in general the machine learning literature typically focuses on subpopulations that are protected by law such as race gender age or religion in practice though there are many subpopulations that we might wish to be fair with respect to and it's not always easy to identify the most relevant ones last summer jen and i along with our colleagues hal dalmae and mira dudik and our incredible intern ken holstein conducted the first systematic investigation of industry teams challenges and needs for support in developing fairer machine learning systems will be which will be published at this year's conference on human factors in computing systems we found among other things that practitioners expressed needs for support in identifying which subpopulations to consider when auditing their systems for biases some practitioners also noted that the most relevant subpopulations may even be application specific with permission i'll read you a quote that illustrates this point from a practitioner whose team develops general purpose machine learning tools people start thinking about sensitive attributes like your ethnicity your religion your sexuality your gender but the biggest problem i found is that these cohorts should be defined based on the domain and problem for example for automated rating evaluation maybe it should be defined based on whether the writer is a native speaker it's also really important to consider intersections of populations the case of emma de graffenreid versus general motors is a really great example of why so in 1977 emma and several other black women sued general motors for discrimination arguing that the company segregated its workforce by race and gender general motors pushed back claiming that they hired black people and they hired women however digging further into the data revealed that all the black people they hired were men who worked in warehouses while all the women they hired were white and worked as secretaries so in other words only by considering the intersection of racing gender was it possible to show that general motors was indeed discriminating against black women one obstacle to detecting and mitigating harms that affect particular subpopulations is access to relevant attributes the machine learning literature generally assumes access to these attributes at the level of individuals however many teams have no such access and instead rely on course grained partial or indirect information one way around this is to ask users to report relevant attributes purely for the purpose of auditing systems for biases however this can raise privacy concerns especially in light of the european union's general data protection regulation or gdpr and some users may object another option is to try to infer the relevant attributes from other available data sources though this can introduce new biases and again users may object sometimes the sub-populations that we wish to be fair with respect to are defined in terms of social constructs which makes accurately measuring relative relevant attributes especially hard for example race is a social construct one that depends on geography context and even time moreover two people can both identify as the same race but have very different physical characteristics therefore in the context of a computer vision system for example it may be preferable to consider fairness with respect to skin tone an easily measurable attribute rather than race as well as being observable skin tone is also more likely to affect the performance of a computer vision system than race per se so before i move on i do want to note that although this talk focuses on fairness with respect to particular subpopulations this is not the only way to think about fairness for example some machine learning papers focus instead on individual fairness or treating similar people similarly and while this is intuitively appealing this approach generally requires the ability to define some quantitative notion of similarity between individuals in practice though this can be challenging to do accurately so we're not going to cover individual fairness here today another way to think about fairness is in terms of counter factuals so for example would i have been hired if everything about me were the same except for my gender again this approach to fairness is intuitively appealing but as isa cola houseman points out it's not possible to manipulate to manipulate race or gender in this way even putting aside practicalities you can't just flip someone's gender and assume that nothing else about them would change for example my personality and characteristics are all heavily influenced by my lived experiences as a woman and so it's just not realistic to expect that i would possess the same personality and characteristics if i had instead been born a man so for this reason we're also not going to cover counterfactual fairness today great so with that i'm going to hand over to jen for the next part of this talk all right okay so hello i'm jen wartman vaughn in the last few minutes hannah's walked us through different types of harm that can arise in machine learning systems and which groups of people are most likely to be at risk from these harms so i am now going to spend the next chunk of time discussing where these harms come from and at the same time strategies that you can take to mitigate these harms i want to emphasize from the very start of this portion of the talk that fairness is not something that can be treated as an afterthought and it needs to be considered throughout the entire machine learning pipeline if you walk away from this part of our talk remembering one thing i hope it's that so a typical machine learning pipeline looks something like this we start by defining the task or problem that would like to solve we next construct a data set data set construction involves selecting a data source acquiring the data pre-processing the data and perhaps labeling the data third we define a model are we going to use a linear model or a decision tree or a neural network what is our objective function each of these choices is associated with its own set of implicit assumptions fourth we train the model on our data we next test and validate the model before deploying the model in the wild and finally we gather feedback about the performance of our model in practice and use this feedback to improve the system we'll see that decisions that are made at every point in this pipeline can introduce bias so let's start at the beginning with the task definition itself what is the problem that you are trying to use machine learning to solve so as one extreme example of what can go wrong here in 2016 a research paper came out by a group in china who were training a face recognition system to predict who is going to commit a crime based on images of people's faces this is extremely concerning for a whole suite of reasons and could lead to substantial harms for people who are misclassified i hope it goes without saying that this is a questionable and rather sketchy task design and i would argue that this is not a task that machine learning should be used for period but there are more subtle examples too consider the problem of gender classification predicting someone's gender from a photo on the surface it might be less clear what are the potential harms here but there are a couple of potential issues that come up so one is that a gender classifier would generally only predict binary gender so it won't work at all for people whose gender is non-binary gender classifiers also reinforce societal stereotypes about how men and women are supposed to look in fact the way that a system like this would probably work is actually by exploiting these very stereotypes it will therefore only work well for people who look like typical men or typical women finally it's often best not to force a gender label on somebody without asking them themselves in the first place there are steps that we can take to mitigate harms during the task definition stage of the pipeline first clearly define your task and the model's intended effects taking gender classification as an example the task here is to classify people as male or female using images of their faces the intended effect is for people who look male to be classified as male and people who look female to be classified as female two try to identify any unintended effects or biases you may need to do some reading here are there known biases for this particular task or domain for gender classification one unintended effect could be that people who don't look like gender norms could be misclassified when thinking about unintent intended effects and biases think about how others might take the output of your system and use this and you know what systems your output may be feeding into three make sure to involve diverse stakeholders and multiple perspectives to try to uncover unknown unknowns and blind spots in your system in our example involving diverse stakeholders might reveal that non-binary people are underrepresented and will likely suffer from poor quality of service here next be willing to redefine the task if you need to or even willing to abort in extreme cases in our example you might first consider adding additional labels beyond male and female but through conversations with diverse stakeholders you may realize that the system actually relies on stereotyping at its very core if misclassifications are too costly you may decide to just give up on the project finally if you do decide to proceed document any unintended effects and biases so that you can keep referring back to them and checking for them throughout the rest of your development process sometimes task definitions change and evolve over time and biases can end up being reintroduced later in the pipeline okay let's move on to data set construction so i'm going to spend a bit more time on this phase of the pipeline than the others because it's very common for biases to creep in here and there are a bunch of different ways that this can happen so one way that this can happen is that the data source may reflect societal biases the world has a lot of bias in it and our data sets usually reflect the world this is what happened when amazon tried to build the machine learning based recruiting tool if your data source contains mostly male resumes and you've historically hired mostly men your machine learning system is going to pick up on this linguistic bias is also a problem people are more likely to say she is a nurse than he is a nurse so as hannah discussed before a translation system trained on text or speech that's generated by people is going to prefer that translation bias can also arise if data is collected from a skewed source so if we train a face recognition system on images of mostly white men then it will likely work well for white men but maybe less well on other populations as another example the city of boston released a smartphone app called street bump in 2011. the app would monitor for bumpiness caused by potholes when people were driving down the street and report these potholes to the city so that they could then be repaired on the surface this seems like a really great idea the problem was that back in 2011 smartphones were much less prevalent than they are today and so the people who use the app tended to be younger and more affluent as a result the potholes that were reported were much more likely to come from wealthy neighborhoods luckily the city did eventually become aware of this problem and quickly acted to counteract it skewed samples can also arise due to the backgrounds and cultural biases of the people involved in the data collection process for example during the interviews that we conducted with machine learning practitioners which hannah mentioned earlier one team reported that they had trouble collecting images of celebrities that were popular in other countries simply because nobody on their team happened to know what these celebrities look like so in order to mitigate harms that arise when selecting your data source you should think critically about potential biases before actually collecting any data check for biases in the data source selection process before building their recruiting system amazon could have carefully considered their data source and realized that it was problematic since there are fewer women in tech jobs and the data would be heavily skewed male try to identify societal biases that are present in your data source we know that there are more female nurses than male nurses and the people are likely to talk about nurses in a gender stereotyped way so this is a bias that again could be identified in advance fourth check for biases in the cultural context of the data source and finally you should check that the data source matches your expected deployment context and this is something that i'll come back to again a little bit later in terms of the process of actually collecting your data once you've identified your source you should check for biases in the technology used to collect the data thinking through potential biases in the technology could have more quickly identified problems with the use of the street bump app check for biases in the humans involved in the data collection process this can catch cases where a team doesn't have the proper cultural background to gather the data that they need for some system next if sampling data points from a large population check for biases in the sampling strategy and assure sufficient representation of subpopulations following these two best practices would have flagged the skewed data sets that many face recognition and gender classification systems were being trained on finally check that the data collection process itself is fair and ethical this one can be tricky even if we know that we need more data from a certain subpopulation there's this question of how we can actually ethically collect it the challenge is to avoid putting a tax on already disadvantaged populations for example by forcing all of our co-workers who are from some minority population to come and supply data for us on every product that we're building yet another way that bias can arise in data set construction is through labeling and pre-processing for example there's a lot of research out there showing that human biases come into play when people are grading essays but some states are still using automated essay grading systems where the training data consists of essays graded by humans this is essentially treating the human scores as ground truth when we know that they're not ground truth once again there are best practices that can be followed in this data labeling and pre-processing stage so first you should check whether you're introducing biases by discarding data for example suppose your data points contain gender what might happen if you just throw away data from anyone who declines to report their gender relatedly you could be introducing biases by bucketing values for example if you're bucketing by race not everyone identifies with only a single race third you should check whether any software that you're using for pre-processing might introduce biases for example you may want to run machine translation on text as a pre-processing step but we've already seen that machine translation systems can do things like swap the gender of pronouns similarly you should check any labeling or annotation software for biases and finally you should consider whether um the human labelers who are in the loop here might introduce bias as what in the case with the automated essay grading many of these data related strategies can be at least partially addressed with better data related standards and documentation this isn't a new idea so other industries have been in exactly this place before including the automobile industry and the electronics industry these days in fact every electronic component ranging from the simplest resistor all the way up to the most complex microprocessor has a corresponding data sheet detailing all of its operating characteristics recommended usage and other information inspired by data sheets for electronic components hannah and i along with some of our amazing collaborators have a project called data sheets for data sets we propose that every data set model or pre-trained api should be accompanied by a data sheet that documents its creation intended uses limitations maintenance legal and ethical considerations and so on we view this as a way to surface potential biases making it easier for teams to develop more fair machine learning systems though it has benefits that go far beyond fairness too this is a prototype of a data sheet that we made for label faces in the wild a well-known computer vision data set consisting of images of faces it's hard to see from the slide but the data sheet contains a variety of information about the data set including a breakdown of images by age gender and so on to help teams construct data sheets for their own data sets we've put together an initial set of questions that cover different types of information that we think belong in a data sheet the questions are divided into seven categories listed here some motivation data set composition the collection process um data pre-processing distribution maintenance and legal and ethical concerns and each of these categories has about five to ten questions for example in the composition category there are questions about the data points themselves as well as questions about recommended data splits and evaluation metrics in the collection process category we have questions about um who is involved in the collection process the time frame over which the data was collected and whether the data is sampled from some larger population the data sheets concept has been pretty well received so far but of course in order to turn it into a standard engineering practice there are a number of implementation changes challenges that have to be addressed i won't go into details here but we and others throughout microsoft are currently working on addressing these challenges and refining this concept for wider use okay so that's enough about data set construction let's move on to model definition so what is a model well a model is just a mathematical abstraction of some part of the world for example we might assume that the price of a house is a linear function of the number of bedrooms the number of bathrooms the number of square feet with a little bit of random noise and variation added by its very nature a model is simpler than the world so choosing a model necessarily means making some assumptions what should be included in the model and what shouldn't how should we include the things that we do decide to include and sometimes these assumptions can privilege some people over others consider predictive policing a predictive policing system may make predictions about where crimes will be committed based on historic arrest data one implicit assumption here is that the number of arrests that are made in a location is an accurate proxy for the amount of crime in that location this doesn't take into account that policing practices can be racially biased or that there may be historic over policing in less affluent neighborhoods as another example it's common for teachers to be evaluated based on factors including their students test scores as highlighted in a federal lawsuit this assumes that students test scores are an accurate predictor of teacher quality but of course this is not always the case and a good teacher should do more than just test prep are also implicit assumptions in the structure of the model chosen so in this toy example in which individuals are represented by two features we've got a majority population and a minority population that look fundamentally different from each other in terms of these two features that we've chosen in this case there's no simple linear classifier that would work well for both this means that you probably don't want to be using a simple linear classifier on these features maybe a non-linear model would be a better choice in this situation or maybe we should just be looking for a different set of features altogether finally harms can be introduced by assumptions that go into the objective function here i'm showing bing image search results for the term boy on the left and the term girl on the right these clearly look very different from each other this probably comes from the fact that bing like all search engines is optimizing for clicks among other things this example shows just how hard it can be to fix fairness issues in different circumstances the word girl may be referring to a child or a woman and users search for this term with different intentions right so in this case for reasons you can imagine one of these attention intentions may be more prevalent than the other so when defining a model we should follow these best practices first clearly define all of our assumptions about the model second try to identify any biases that are present in these assumptions the ability to use number of arrests as a proxy for amount of crime is an assumption and should be questioned third check whether the model structure introduces biases next check the objective function for any unintended effects we're making an assumption when we design a search engine or a website to optimize clicks and assumptions like these should be questioned in the context of whether they might introduce biases you might also consider including some notion of fairness directly in the objective function if this is something that it's appropriate to do in your setting okay let's move on to the training process so the training process is the stage of the pipeline where we optimize or learn the parameters of a model the weights w1 w2 and w3 in this example that i showed earlier based on the training data that you have in your objective so there's some good news here once you've settled on your data set your model and your objective the actual training algorithm that you use is probably not going to introduce any additional bias we see this as a common misconception actually you generally don't have a biased algorithm at least not a bias training algorithm the problem usually comes more directly from the model you've chosen or the objective or these other factors that we've discussed okay so the testing phase of the pipeline is your opportunity to check for biases and potential harms and problems can arise if you don't have the right testing data or the right metrics in mind for example let's return to this research paper that showed that commercial gender classification software performed poorly on women with dark skin this is the kind of problem that can be caught during the testing phase if you want to build products that work well for everyone then you should explicitly be testing products performance on different segments of the population in the case of the compass recidivism prediction tool there was some internal testing but external audits still revealed unfairness in this system so to talk about why we need to get a little bit into metrics i'm going to spend just a couple of minutes going through some different metrics here just to give you an overview of what the landscape looks like fairness metrics are currently most well-defined for classification problems to simplify things i'm going to focus only on binary classification in my discussion here and the main thing that i want you to take away from this is that there are a bunch of fairness metrics and these metrics are more or less appropriate in different circumstances if you want to learn more there's a great tutorial by arvin narayanan from fat star 2018 that you can find online okay so to define the fairness metrics that are typically considered for binary classification it's useful to start with the idea of a confusion matrix which many people might already be familiar with here i'm showing a confusion matrix for a hiring scenario we have some applicants who are qualified for a job and some applicants who are unqualified our machine learning algorithm chooses to hire some of these applicants and to reject others qualified applicants who are hired are true positives while unqualified applicants who are rejected are true negatives your algorithm is getting both of these right meanwhile qualified applicants who are rejected are false negatives and unqualified applicants who are hired are false positives the confusion matrix simply tells us how many applicants fall into each of these four buckets and these four numbers that are given by a confusion matrix can be used to compute various standard machine learning performance metrics like accuracy and precision so rather than considering only a single confusion matrix for a whole set of applicants we can choose to break things down by demographic and compute separate confusion matrices for say our male applicants and our female applicants to be concrete we filled in one example of what these matrices might look like with some numbers here notice that in this example there are 100 men and 100 women in total for both men and women the system makes correct decisions on 75 out of 100 applicants 60 qualified men are hired and 15 unqualified men are rejected these are all correct decisions meanwhile 15 qualified women are hired and 60 unqualified women are rejected also correct decisions so the accuracy is the same in both of these cases but the system makes different types of errors on these two groups so let's see how a few different fairness metrics play out in this case so the first fairness metric that you might hear about is demographic parity demographic parody is based only on classifier output and doesn't take into account true labels at all it says that applicants should have about the same probability of being hired regardless of whether they're male 1 or female so the ratio of hired applicants to total applicants should be the same for each group or at least approximately the same for each group if you happen to have heard of the eeocs 80 20 disparate impact rule this is a very similar idea so in this case the system does not satisfy demographic parity because 80 of 100 men are hired while 20 of 100 women are hired but you know maybe you might say this is okay because in this particular example more of the men are qualified than the women um just as an aside demographic parity is perhaps the first metric that you would want to look at when evaluating an automated hiring system like the one that amazon proposed so going a bit deeper predictive parity takes the true label into account as well as the classifier output it looks at the probability that an applicant from some group is qualified given that you've chosen to hire this applicant and it says that this should be the same or about the same for both of these groups in this case the system does satisfy predictive parity since for both of these groups three-quarters of hired applicants are qualified now false positive rate balance is another metric that says that false positive rates should be the same for both groups that is the probability that an unqualified applicant is hired must be the same or about the same for men as it is for women in other words your classifier should treat all unqualified applicants the same regardless of whether they're male or female in our example the false positive rates are different with a substantially larger fraction of unqualified men being higher than unqualified women false negative rate balance is essentially the same thing but for false negative rates it says that the probability of a qualified applicant being rejected should be roughly the same as for men as it is for women in other words the classifier should treat all qualified applicants the same regardless of whether they're male or female again in this example our false negative rates are different with qualified female candidates much less likely to be higher than qualified male candidates you may sometimes hear the term equalized odds come up and in the context of binary classification this simply means that you simultaneously satisfy false positive rate balance and false negative rate balance basically treating all similarly qualified candidates similarly so this notion of equalized odds that i just mentioned is effectively the fairness metric the propublica considered when it was auditing the compass system for racial bias in other words they asked whether compass makes similar errors in terms of both type of error and quantity of error for black and white defendants and indeed they found that it does not because of this they said that the system was racially biased in response northpoint the company that built compass argued that compass does satisfy predictive parity and is therefore fair but it's so there's a lot of back and forth about this and about why the system couldn't just satisfy all of these metrics but it turns out that things are not so easy in fact it turns out that it's mathematically impossible for a system to simultaneously satisfy predictive parity false positive rate balance and false negative rebalance any system that satisfies two of these properties must necessarily fail to satisfy the third this impossibility theorem was really at the heart of the public debates around compass and rachel bias as long as north point requires the compass satisfy predictive parity it can never satisfy equalized odds so what should northpoint do in this situation well this is a tough question but people have made the argument that given the impact of these decisions on people's lives equalizing false positives and false negative rates between races is more important perhaps than calibration so equalize odds is arguably a better measure here okay so what are some best practices for testing first you should ensure that the test data set matches the context in which the system is expected to be deployed and ensure that the test data has sufficient representation in it doing these things would have helped developers identify the performance imbalance and gender classification systems before these systems were ever deployed in the first case um of course all the standard risks of overfitting apply here so i just want to mention you should be aware of this if you find yourself really relying on one particular test data set and tuning your model to be fair on one particular test data set next you should involve diverse stakeholders who represent multiple perspectives to ensure that you're testing for the right things the diverse voices project from the tech policy lab at uw has created guidelines for making tech policy more inclusive by having short targeted conversations with panels of people who have diverse viewpoints similar ideas could easily be used in product development as well additionally you should clearly state all of the fairness requirements for your system and use appropriate metrics to ensure that your requirements are met but of course you should keep in mind here that there are trade-offs that need to be made with any of these metrics and that again many aspects of fairness are just not possible to capture with metrics at all as hannah mentioned before fairness is a socio-technical concept and we can't always use quantifiable metrics to capture it okay so moving on to deployment the most common issue here is the one that i mentioned briefly a few minutes ago that the deployment population is somehow different from the population that you either implicitly or explicitly assumed that you would have that is your deployment population is different from the population from which your training and testing data were generated or the population that you had in mind when you were defining your model a common example is collecting training data from people in one country say the us and then deploying a system in other parts of the world there's actually some interesting research way back in 2011 they looked at available face recognition tools and showed that the location where a face recognition system was developed had significant impacts on its performance on different populations specifically systems were substantially more accurate on faces from the same geographical region that the system was developed in when deploying a system you should double check that the data source matches the deployment context and keep an eye out for discrepancies as you collect more data on your system's performance you should also monitor your fairness metrics for any unexpected changes at this point in the pipeline it can also be advantageous to invite diverse stakeholders to audit your system for potential biases for the initial audits it's really better to be in control of this process yourself you don't want to be in a situation where someone from the outside audits your product for fairness before you get a chance to remember it was external audits that revealed that commercial gender classification software had different performance on different groups finally you should monitor user reports for any potential fairness issues all right so finally there's the feedback stage this is something that's discussed a lot in the context of predictive policing and hot spots so as we've already discussed predictive policing systems operate under this assumption that more rest in an area means more crime this can create a feedback loop or a self-fulfilling prophecy more officers are deployed to neighborhoods where more crime is predicted and this ends up leading to more arrests which leads to higher crime being predicted in the future and to even more officers being deployed to these areas adversarial feedback can also be an issue here a few years ago when microsoft released the social chatbot tay there are groups who took advantage of tay's use of feedback in order to adversarially insert biases and other harms into the system the best practices here at the feedback stage overlap heavily with those for the deployment stage which makes sense because the line between these stages can be pretty fuzzy you should continue to monitor the match between your training data and the data that you're seeing in the wild continue to monitor fairness metrics and continue to monitor user reports too monitoring appropriate fairness metrics could surface that predictive policing systems are getting more biased over time you should also monitor users interactions with your system and if necessary you might consider prohibiting some types of interaction altogether for example for a chatbot like tay in addition to monitoring things it may be necessary to actually restrict certain types of feedback especially because in this case the severity of the harms that can arise is high all right so we have made it through the machine learning pipeline we've talked about both where harms come from and strategies that you can take to mitigate these harms and with that i'm going to turn things back over to hannah all right so for the last part of this talk i'm going to tell you about some recently developed software tools for detecting and mitigating harms over the past few years the academic machine learning community has turned its attention to the topic of fairness and machine learning and while this graph here isn't strictly accurate over the past few years we've seen a massive increase in the number of papers published on this topic spurred by this increase we've also seen researchers and practitioners start to develop software tools that implement the ideas put forward in these papers so i want to run through a handful of these tools with you just so that they're on your radar okay so the tools i want to tell you about fall into a couple of different categories the first category is auditing both auditing data sets to uncover potential biases and auditing the outputs of systems to see if those outputs are fair according to the kinds of metrics that jen told you about earlier the first tool i want to highlight is equitass which was developed by researchers at the university of chicago's center for data science and public policy equitas consists of a web audit tool a python library and a command line tool for auditing the outputs of systems for allocating opportunities resources or information according to some of the metrics that jen told you about previously plus a couple of other metrics as well it's pretty easy to use and generates really nice bias reports that is a little limited in what it can do the second auditing tool that should be on your radar is ibm's ai fairness 360. this tool was recently released and received a lot of attention in the media like equitas it's focused specifically on systems for allocating opportunities resources or information and it does implement more fairness metrics than equitas though it's not entirely clear how commonly used some of these other metrics are one nice thing which i haven't played around with yet is that it implements some metrics for assessing individual fairness unlike equitas before i move on to the next category of tools i do want to again emphasize that fairness is a fundamentally socio-technical challenge so these tools are not be-all and end-all solutions and are only appropriate in certain particularly limited circumstances specifically they focus on harms of allocation and certain types of quality of service and not on other types of harm moreover there are many aspects of fairness such as justice and due process and so on and so forth that aren't captured by metrics that look only at parity in decision making and finally as jen told you it's impossible to simultaneously satisfy all of these fairness metrics except i