Task-Oriented Dialogue as Dataflow Synthesis

Key Takeaways

hey what's up my calendar today your tackle talk started eight seconds ago already is jason free to help between now and the end of the talk yes he's free now invite him and his manager right away okay i've invited them actually invite the whole team okay i've added 42 other people to the meeting would you like to join let's do it you are now in the talk hey what's this talk about anyway it's about representing task-oriented dialogues like this one so supporting natural spoken interaction turns out to be really hard to explore this problem we collected dialogues using wizard of oz we're releasing a large data set which we'll show you later we found that when you give users freedom they have all kinds of language behaviors these task oriented dialogues didn't follow any set script to annotate them we needed a way to represent these natural conversations slot filling wasn't enough because our users didn't just answer who what when and where they often requested computations of new values or referred back to existing values like dance and the talk or they revised old computations like when dan changed his mind and said invite the whole team to support the full range of user requests we represent the dialogue state as a data flow graph a history of all the grounded computations that are performed during the dialogue some computations are new while others reach back and retrieve or transform earlier computations there are 45 authors on this paper researchers engineers designers and data specialists and we've been having a lot of fun working on this we hope you'll enjoy watching the team talk through key aspects of our approach more formal details around the paper dance conversation at the beginning of a talk started with a simple question what's on my calendar today that's easy we've known for decades how to build command and control system for formulaic inputs exactly list out all of the intents and slots for each intent then just learn how to predict those things from the user utterances but that will break down pretty quickly have you ever seen this intent before i mean what is this intent is x3 between now and the end of y those are weird slots it's not really one intent anymore now the dialog agent has several tasks it needs to figure out who jason is look up the event compare times on a calendar and then say the right thing back to the user these all have to link up somehow the semantic parsing literature does have a huge body of work about predicting descriptions of complex and compositional computations but how do we turn a semantic parser into a general purpose dialog system well dan's request is naturally formulated as a program which we can represent as a data flow graph a dialog agent just needs to predict a graph like this after every turn but the graph isn't a representation of the meaning of the user's utterance instead it's a representation of what the agent should do in response right so the structure of the agent's program may not mirror the syntactic structure of the user's request there will be discrepancies although we should try to minimize them to make prediction easier hang on dan didn't actually say the words my tackle talk he just asked about his calendar and it was the bot that mentioned the talk so in dan's next question he could simply refer to it as the talk so i don't think these requests are independent these programs are chained together somehow yeah the two speakers are cumulatively constructing a more complex computation it's really collaborative programming through dialogue the speakers can each extend the data flow graph bringing new entities into play or encouraging the other speaker to do so in fact that's why it's useful to represent programs as dataflow graphs we're just reifying all the intermediate computations as graph nodes that can be referenced later in the dialog eventually maybe the nodes can include shared goals and plans hey guys this could really help language generation too each data flow node records how it was computed so the agent can spell out how it got its answer this means that we can say complicated new things just like we can understand complicated new things we just need to express the right level of detail that's awesome it's going to need its own talk though okay so back to language understanding what actually happens when the user asks when does the talk end how do we connect that back to the previous graph this isn't so hard right if semantic parsing is about synthesizing computation graphs dialogue should be about transforming computation graphs we need a notion of meta computation a regular computation node goes and queries things like an external database but a meta computation node can query the dialogue history itself so a phrase like the talk predicts a meta computation node that goes ahead and finds a previously mentioned talk why not just directly predict which old nodes to copy a lot of contextual semantic parsers use a copy mechanism a copy mechanism might handle some of this but the values you want aren't always in the dialogue history let's think about another example what's the weather like means that you want the weather here and now by default but when some time and place are already under discussion you want the weather then and there if you take a step back what's the weather going to be like is always a question about the most salient time and place it's just like when you use a name or a pronoun which laura did you mean well the laura that you were talking about but if the agent can't find any laura in the dialogue history it's probably the lore that you know best which would come from a database okay so let's package this up as a matter computation operator we introduce a data flow function called refer it finds the most resilient thing that matches some description this makes the learning problem easier it's not easy for the dialogue agent to predict a program that looks up the most salient person named laura that search depends on both the dialogue context and every interaction you've ever had with someone named laura but now laura predictably turns into a new dataflow node that says find a salient laura and we can train a separate salience model to make that decision and here we can take advantage of a huge body of existing literature on tasks like correference and bridging an offer reference resolution is really hard so it's good to separate it out david said earlier that dialogue is about transforming computation graphs i see that our semantic parse can use the reference operator to query the dialog history just like querying a database but you can also modify a database so how about modifying the history this is also important users change their mind all the time like when dan invited the whole team for utterances like this we need to edit a note's parents or maybe it's grandparents that's another kind of meta computation called the revision operator when you evaluate it decides how to update the old computation and re-execute it wait don't try to rewrite history keep the old nodes around they're still salient users might still refer to them with pronouns so we need a non-destructive revision operator it'll just add an updated version of the old computation copy on right so the graph will still remember both versions actually this revision mechanism seems really powerful is there anything else we can use this for other than users who change their mind sure how about users who make a mistake or users who leave something out either way the program will throw an exception so the bot has to complain or maybe ask a polite question great now when the user reacts anything that looks like an answer in this context is probably a request to fix the graph and we can learn that from training data yeah it's not like the dialogue state manager is in a state that demands an answer the user says whatever they want and we predict what we should compute in response right sometimes we'll predict a brand new intent but if the user was only providing a new constraint we should predict a little program that will revise it into the correct place and then we'll run that revised program these two meta-computation operations are obviously not enough to compactly represent all possible discourse phenomena but as we'll see they seem to cover a large fraction of the cases that come up in real dialogues so in summary everything that happens is added to the data flow graph it's a record of the dialogue history all of the computations that were predicted in response to the user which include things they asked for things we did and things we said back to them the dialogue agent is just a learned model that looks at the current data flow graph and a new user utterance and produces new nodes to add to that dataflow graph these new nodes could be new values new functions of all values references to old values or revisions of old computations won't these graphs get big yeah they can let's take a look so to train our dialogue agent we'll need a supervised data set conversations about real stuff where each user utterance is annotated with the subgraph we should add in response where do we get this data don't worry we've got data to evaluate this approach we collected a new data set called sm cal flow consisting of more than forty thousand dialogues between a user and an agent each turn in this data set is annotated with a dataflow program we put crowd workers in over a hundred scenarios to play the part of the user we had more experienced wizards of oz steer our systems response choosing what it says and why conversations involve diverse complex user goals and lots of cross-domain interaction there's lots of hard stuff in sm cal flow that doesn't show up in existing data sets like users who are unfamiliar with the system's capabilities overall this is the largest and the most complex task oriented dialog data set currently available what can we really use these data to learn to do all those complicated things we were talking about a couple of slides back we can a basic seek to seek model improves by ten percent when we add our meta computation and operators to a copy based baseline and it doesn't just help on our data set in addition to sm cal flow we're releasing a meta computation version of multi-was where we see similar improvements of course there's a ton of interesting structure in this problem that c2c doesn't know about and we could use pre-training or multitasking or indirect supervision i wish we had time to get into that today yes this is a rich problem and we certainly haven't solved everything audience if you are interested in exploring the sm calf flow data set we have set up a leaderboard where you can beat our business model and compare to other submissions we made it i get to relax now right the next event is a talk at the interactive and executable semantic parsing workshop on november 19th after that you have generation paper planning i guess we'd better get started oh could you please show a picture of all the amazing people who worked on this of course thanks for watching everyone