Stanford Seminar - Human-AI Interaction Under Societal Disagreement

May 19, 2023 Mitchell Gordon of Stanford University Whose voices - whose labels - should a machine learning algorithm learn to emulate? For AI tasks ranging from online comment toxicity detection to poster design to medical treatment, different groups in society may have irreconcilable disagreements about what constitutes ground truth. Today's supervised machine learning pipeline typically resolves these disagreements implicitly by majority vote over annotators' opinions. This majoritarian procedure abstracts individual people out of the pipeline and collapses their labels into an aggregate pseudo-human, ignoring minority groups' labels. In this talk, I will present Jury Learning: an interactive AI architecture that enables developers to explicitly reason over whose voice a model ought to emulate through the metaphor of a jury. Through our exploratory interface, practitioners can declaratively define which people or groups, in what proportion, determine the classifier's prediction. To evaluate models under societal disagreement, I will also present The Disagreement Deconvolution: a metric transformation showing how, in abstracting away the individual people that models impact, current metrics dramatically overstate the performance of many user-facing tasks. These components become building blocks of a new pipeline for encoding our goals and values in human-AI systems, which strives to bridge principles of HCI with the realities of machine learning. About the speaker: Mitchell L. Gordon is a computer science PhD student at Stanford University in the Human-Computer Interaction group, advised by Michael Bernstein and James Landay. He designs interactive systems and evaluation approaches that bridge principles of human-computer interaction with the realities of machine learning. His work has won awards at top conferences in human-computer interaction and artificial intelligence, including a Best Paper award at CHI and an Oral at NeurIPS. He is supported by an Apple P