Research talk: WebQA: Multihop and multimodal

Key Takeaways

hey thank you for joining us at the microsoft research summit 2021 i'm jonathan bisk from carnegie mellon university and today i'm going to be talking about a new data set that we've just released as part of a nurip's challenge web qa multi-hop and multi-modal question answering this is joint work with a number of collaborators around microsoft research and the lead author on this is my student ying cheng who i'll reference again later in the presentation so what are we doing here and what's sort of the point of this this kind of data set and introducing a new challenge so the idea here is that the way that we access the web when we're using it is that we'll type questions into these little boxes and we assume that some magical engine is going to be able to find the answer but that answer may not rely in text it may rely on images or some other modality so we're trying to take one step towards moving models towards the kinds of web experiences that humans have so let me make that more concrete i have a question like are there picnic tables in chenle park that's a local park here in pittsburgh so if i go to bing and i search that i'm going to get a couple of answers so i might get this first one here saying that there's a some sort of blanket and board in pittsburgh is going to have some sort of a picnic maybe in schenley park doesn't answer my question i might get this other thing about the parks department website which is talking a little bit about some of the facilities uh the fact that you can have picnics there still not answering my question so i've got these snippets which are you know semantically related they're relevant except that they're not actually what i'm looking for and as a human we sort of already would have guessed that when i told you i was going to ask the question are there picnic tables you probably don't even look at these sets of search results you don't even look at that main page of text because you know that there's this other tab that will just give you the answer immediately so here we see it the first thing is someone's picture of a picnic bench in shenley park we also see some maybe picnic tables which are not helpful so it's not the case that every image is going to answer our question but it is the case that some answers are just more readily accessible if we have access to a second modality if we have access to in this case images and so then what we want is that we eventually have some sort of a search engine we have bing or whatever it may be that basically takes in that question aggregates the knowledge that's available to it across both of these modalities and then produces a fluent answer so yes there are uncovered picnic benches in schenley park now there may also be covered picnic benches but we don't have evidence for that here and so this is the kind of reasoning that we would want it to have it's sort of correct it's multimodal and it potentially uses multiple sources and and cites it and cites them as part of that now i also mentioned multi-hop in the title because there's also more complicated questions that we tend to see or we might want to be able to ask a system so for example what type of currency was used during cleopatra the eighth's reign it's a somewhat specific question you can imagine these are the kinds of questions that children that are trying to cheat on their homework will tend to put into a search engine and so what we need is we do get relevant information from bing so here we get something about telematic coinage which is actually going to be the answer but there's nothing here that specifies cleopatra we then get this other source that tells us about previous rulers and then a reference to this last queen cleopatra and that she was during this age that dynasty overlapped and so what we need are both of those pieces of information to be aggregated so that we can actually produce an answer in this case just the simple phrase um solemn ptolemaic coinage and so the idea here is that the kinds of things that you and i do when we walk up to a search engine are take a bunch of snippets take a bunch of images we have some preconceived notion of where the answer probably is going to be and then some way that we might reconcile pieces of information in order to get the correct full answer and we just want systems to be able to do that as well right now the literature is largely split so we have people that are focusing on these kinds of multi-hop questions in the text domain we have people that are working on increasingly strong visual uh challenges but we're not seeing something that's at that intersection that's doing the same thing that we're doing aggregating across both so what's the difference between questions that are being addressed to for instance images versus to text well if we look at kind of a word cloud this is from the data set that we've collected you start to see things about libraries artwork churches cathedrals attractions these are places you know people places things stuff that you see when you're out and about in the world very different from clinical psychology natural history these um slightly more maybe academic topics that are sometimes present in the text corpora and so again in the same way that we know that if i'm looking to see if there's a picnic bench an image is going to quickly answer the question we want our systems to be able to do that to be able to reason across both of these types of knowledge so how do i know that let's say people are interested in these kinds of questions well i know that they are because i ask these kinds of questions i do this kind of reasoning and everyone i've spoken to has fired off at least one query in the last week or month where they didn't think it was going to be that complicated the system didn't answer it correctly and they had to do a little bit of digging the answers were there but they had to find them for themselves so we do know that people are answering the asking these questions of web search but they're not doing it at scale so and the correct answer in some sense to do people ask these questions of web search is no because while they're there they're very rare we've all been trained out of it we've all been trained that this question is probably a little bit too hard and so i'm not gonna i'm not gonna expect anything of it and instead we put keywords in and then try to do multiple searches to aggregate the information and the way that we know this as i was saying is this is rare is that we actually went and looked at the bing logs so without going into too much more detail here we know both from those logs that people are asking these questions and we also know that it's pretty rare that they're actually either getting an answer or in general that they you know follow up with these kinds of things because they they get sort of disheartened by the fact that there really just isn't the kind of response that they wanted when they thought it was probably something simple so we have to address this problem we can't just go and grab a bunch of logs and then you know answer those questions we instead have to collect these questions artificially we have to construct a scenario so that we can train systems so that in the future we don't have this problem in the future we can deploy systems that are able to do this kind of complicated reasoning this cross-modal reasoning these kind of full sentence kinds of answers that you would want from an assistant and that way when people come to a search engine they actually ask more of these things we get a nice sort of feedback loop because they're getting some of what they want and so then they give us more data to increasingly sort of build robust systems so how did we collect the data so what we did was we had a bunch of users on mechanical turk and this is for an image example here that we showed them topically related images so these may be based off of the same location or the same sort of type of structures or things like this users actually also annotators actually also proposed possible clusters that would be useful for us and then we asked them grab one or two of these images and ask a question that would sort of require information aggregation across them so here you're seeing a simple interface they choose one two three four five or six they write some sort of a question and they provide us an answer now we want to make sure that what they're providing us is correct so we have a pretty rigorous validation process which i'm not going to go into in too much detail here but basically there's a sort of a correctness and an agreement aspect to it so we have people go through and say like are these images that are being used are the facts that the person is relying on actually valid are these accurate are these you know do they correctly merge to answer the question and then we also run another validation where we have people given the question and those same sources produce their own answer so we can see whether or not they're producing very similar sorts of answers so we can get to a place where we have data that looks very much like what a human would want so how much data are we talking about so i'm going to break it up first by modality so in terms of questions that rely on images we're looking at an overall here just shy of around 20 000 examples same in the text you see these are split up predominantly in train and then we have a smaller dev and a larger test set so overall about 34 000 of these pairs for training sets the sources the question the answer and then 5000 for development and seven and a half thousand or so for testing the other thing is we want to make sure that there's no sort of obvious sort of artifacts in the data that people are going to be able to have their models pick up on and so what we're doing here is in the same way that when you show up to a website there's maybe the two facts that you need and eight facts that are irrelevant we also are going to parallel that with distractors in both modalities so you have an image the questions tend to be about 23 words on average the answers maybe take about 21 words on average and then you have two sources of additional text so every image comes with captions and all text is a snippet and so we're getting we're going to have sort of correct captions that are almost exactly the same length distributions as incorrect captions um then similarly for the text side we have this kind of thing with snippets being uh distractors that are sort of parallel in length and you see the questions are similar in length between the two modalities the answers are slightly different so overall we're not seeing anything where there's a clear giveaway that like oh there's a few more words here it's probably the correct answer that tends not to be the case and so we don't have those kinds of spurious issues in the data the other problem that we had to address in this work is how do you evaluate so so far we've just been saying like okay very intuitive paradigm this is something you and i do all the time uh we're going to have our systems do something very similar they're going to go through the facts they're going to give us an answer but that answer is natural language text and if you have a system which is producing a full sentence there's a whole different research agenda there on just how to do evaluation of nlg systems so the way that we try to address this without sort of taking on that full challenge is breaking up the evaluation into two pieces we have what is sort of a fluency score with a little bit of semantics mixed in we're relying on bart score for this and then we have an accuracy where we actually had annotators provide sort of what is the noun phrase what is the key entity that answers this question or maybe just the the key property so you know a color and so we can check for those we can also make sure that they're still producing fluent and coherent language and then we can simply multiply those two scores together in order to give a sort of overall sense of how well a system is performing so that's what the numbers are that you're about to see on the next slide so we have how well can a model retrieve the right sources versus a human humans are much better at it how well can a system perform if it has predicted sources and how well can a system or a human perform with sort of the ground truth sources and the only thing i want to draw your attention to here very quickly is that while fluency really you know pretty impressive um accuracy really substantially uh different and so there's definitely more that can be done in terms of language generation make it sort of more natural sounding but this is really the focus of this challenge and that's also why you end up with this really big overall gap between humans and systems so i wanted to kind of give you a few other little quick teasers before we close so this this gives you kind of a layout there's an interactive version of this you can play around with of the types of questions that people are asking but the important takeaway is that you have these kinds of questions broken up into a bunch of different semantic categories do they do the systems answer yes no questions well can they choose between contrasting options well can they handle color questions things like this and you see that certain tasks are much harder than others none of them are in general all that hard for humans so in particular i think this is not going to come as a surprise to anybody but shape and number can you reason about counts can you reason about inequalities this is something that the systems that we currently have are just completely incapable of doing they're looking to basically copy an answer from the text they're not looking to reason but you have to do a certain amount of reasoning if you're going to do multi-hop you have to be able to aggregate information and knowledge and produce something new the final column here text base so these are the ones where we don't have sort of a clear sort of maybe sort of finite set of answers so yes and no being a finite set of just two and so these are places where we're talking about sort of open domain entities and we see that there's still a very substantial gap here so in terms of knowing which entities in a snippet are related to the question maybe the systems do an okay job how those entities relate to each other relate to properties of one another still have quite a bit of ways to go so where does that leave us well it leaves us with a pretty long road ahead so this is just one quick failure case that i thought was pretty funny we have what is the sculpted bust at the baroque library prague wearing on its head and then we provide the image so step one figuring out that this is the bust that we're talking about the systems are given access to some metadata that really shouldn't be a huge problem for them it's something that even without metadata humans don't really have a trouble figuring out if you have a contrasting set of things that are not busts um but what we see is that the model has said very fluently the sculpted busk bust at the baroque library prague is wearing a helmet on its head and why helmet because that's a natural continuation it's a totally fluent thing the correct answer is obviously a flower wreath so visual understanding still still still going to take a while anyway if you want to learn more about this the paper can participate in the challenge the website is here there's a leaderboard for you to participate and again huge shout out to my student chang who's actually the lead on all of this collection thank you