Build Chatbots Faster with Large Language Models

Key Takeaways

thank you [Music] hello this is Dr Rachel tatman for cohere and today I want to talk about some tips and tricks for helping to use large language models to build chat Bots a little bit faster and the first thing that I want to bring up is that I would strongly advise against serving raw generated text to users for both ux and security reasons right it's going to be unpredictable and also large language model output by default is not grounded is not tied to any information so if for example you have a Commerce chat bot and someone attempts to order something you may get a chat bot a large language model producing output like order placed successfully even when that order was not successfully placed so I just recommend it for that reason and also for security reasons a lot of the adversarial attacks against large language model based systems require access to Raw text output so if you don't have that you just want to do with the adversarial attacks so given that I wouldn't recommend that what would I recommend in order to increase the pace of chatbot development so my general recommendation is human in the loop data augmentation which means using the large language model to generate additional data that is human validated that allows you to get a little bit of a warmer start when you are training or fine-tuning a chatbot so when is it especially helpful to do data augmentation well first of all you may not have represented representative data for all of your targeted user personas so you may be building a chatbot for example this Commerce chat bot and you may be targeting a couple of different personas maybe someone who's doing their weekly grocery shopping and perhaps someone else who is shopping for a party and you want to make sure that both those users get their needs met and if you don't have representative data for both of those people a language large language model can help you to to flesh out your your initial training data um you can also use it to generate examples for new specific intents for example if you have a new topic that suddenly becomes relevant and you need to add it to your chatbot you can use um you know large language model generated data to help you get going a little bit faster there especially if you can't find existing data for it and finally if you have data but it is too clean or it's not representative of user generated text so uh most research data sets for example tend to be very clean right they tend to be you know have consistent capitalization and punctuation they tend not to have a lot of spelling errors so being able to generate Text data that is a little bit noisier can be very helpful if that's what your system is going to see in production so uh if you know that you need to do data augmentation uh why would you use large language models over you know templatic data augmentation or perhaps a strategy like translating to and from another language so one of the biggest benefits is you can avoid the repetition in a template based data augmentation so template based data augmentation usually has for example more or less the same syntactic structure and you can sometimes get a sort of weird um depending on how you're doing that augmentation weird Edge errors where a human probably wouldn't say that so in an idiom like turning the page turn is often used as a synonym for rotate but of course they are not they're not replaceable here rotating the page means something different from turning the page it can also if your other option was to generate all new data which you know always an option um generally going to be a little bit slower and a little bit more expensive and perhaps a little bit less easy to tune and a final reason why it makes particular use to use a large language model is because if you are going to be encountering noisy user generated text well large language models are trained in no small part on noisy user generated text so for example uh the the cohere large language models is here is from the documentation is trained on among other things common crawl which includes a lot of social media data and we have some links here to some of the different sites that are highly included and they include things like Tumblr and stack exchange medium which has comments as well so you have existing examples of noise use or generated text and the training data which means that the model is more likely to be able to successfully generate examples of that for your work so that's why you would do it uh how to do it and of course depending on your specific case you may have a lot of variation in what you actually choose to do but here are some recommendations um so uh I would recommend working with what existing data you do have and using that to Prime the model doing prompt engineering to generate new data so we have an example that we're going to be working through here I've taken some intents from the slurp data set which was developed by bastianelli at all 2020 which is uh freely available and the prompts here were generated by mechanical work a Mechanical Turk workers and the transcribed data is very clean and fairly formal so here are some examples for the intent play music the intent labels here are generated by me so some example utterances would be play my favorite playlist play Irene by Toby Mac start my jazz playlist shuffle and play the playlist so if this was our existing data and we wanted to get more training examples for our chat bot how could we do this so one way to do it would be to prompt using this existing data and also including a little bit of formatting in your prompt so for example here I'm going to use this heading intent colon name of the intent I'm also going to include a markdown list format with dashes at the beginning of each line and then I will include at the end a dash that is empty after it to uh prompt the model to continue working on the list so uh just an example of what that looks like this is from the cohere playground you can see here using this Paradigm uh I got a bunch of additional examples um that more or less were usable so uh of these examples uh one the only one that was generated that I would say definitely doesn't belong in this intent is set up the alarm setting an alarm is a different task from playing music whereas other examples like start shuffle my music resume playing playing this album plays jazz in the kitchen uh I would say are all you know perfectly acceptable and the last item looks like it was maybe cut off uh resume the song that I was which I would say is not a good training example but is more or less in this intent so about 75 of these examples I would be willing to include in the training data for this just to do another example so this is a separate intent all these examples here are human generated uh and the intent here is set an alarm or reminder um so put an alarm on my calendar every week I eat a 6 a.m wake-up call and using the same prompting um Paradigm we got about 80 of the generated examples were usable um so the one that is a little bit questionable is ring an alarm to notify me when something will happen um I when something will happen is not uh not particularly tractable as a a Target time to set an alarm but the rest of them I would say are pretty good training examples right so example add a reminder to my calendar every Monday to read news so these were all uh ways of increasing the volume of data you had in a way that was very similar to the existing data what if you wanted to add some data diversity um so some techniques for prompting that I think would be a little bit more risky would be to prompt based on emotion or prompt based on a stated specific user Persona and something that I think is a little bit less risky but is going to require more domain knowledge is prompting by referencing specific websites so uh prompting based on emotion um has a couple different problems so here I have an example of uh someone angrily asking a chat bot to play music uh and the first example I want to listen to music now capitalized I'd say fits the prompt but as we go down through the existing prompts through the generated data you can see that the uh the intent times to actually switch so some of the generated examples are stop playing music stop playing that music music off which is the opposite of the intent that you are intending uh and part of the issue here is that emotional context and intent are not IID right are not independently distributed so if you are have an intent for booking a flight uh you are probably going to have more neutral sort of emotions associated with that uh if you are talking to customer service after your flight has been canceled and trying to rebook you're much more likely to get Negative emotions with that so attempting to prompt for intent uh to generate data for intents based on emotional content is likely to often lead to difficulties and generated text that doesn't fit in your intended intent it may be situationally useful if you're trying to create data for an intent that is often associated with a strong emotion especially a strong negative emotion uh on the other hand you might think to try user-based prompts right so here we have examples of a 20 year old student asking a chat bot to stay mute to play music and you can see that the examples are fairly generic or fairly similar to the um the prompts that was not uh primed with a specific user Persona um but uh one thing that I would probably point out here is one of these examples is play the latest hits um I don't know that I've ever heard a 20 year old student say anything uh like that what I have heard is uh people who are not in that group stereotyping that group by their love of the latest hits so the issue here is that if you are specifying the type of user that you want stereotyping is very likely and this is due uh in part to the underlying distribution generally people don't introduce themselves by their demographic qualities in an online Forum if I am doing that usually I'm doing it to stereotype or joke about those people right so for example this would be like if I were joking about people from Ohio I might say ah look at me I'm from Ohio I love the Ohio State um when that is not I would say and especially Fair characterization of all people from Ohio um and I would say one possible exception here would be to try and get multilingual data so here we have an example of attempting to get similar um data except it is in French um and I would say the success rate is okay uh one of the generated um uh one of the generated items uh means something completely different so check the check box instead of play music um and one of the items is in the same general topic so this this top one I believe a balladua is a Walkman which again probably not something a lot of people are talking about uh when prompting a voice assistant so potentially useful but proceed with caution what I think is probably going to be more generally useful for most folks is to use social media sites as proxies for personas and um domains so they are interested in so uh just an as an example uh there is Reddit data in this data set and Reddit tends to have a very strong skew towards male users so it has about twice as many uh male as female users or if you really wanted uh you know to focus on younger users the majority of Snapchat users are under 29. um so obviously different social media sites also have an effect of topic so very specialized platforms are more likely to have very specific types of language used on them so next door is used for talking about local matters stack Overflow used for um talking about you know technology and programming Ravelry is a Fiber Arts based social networking site again very narrow domain um the benefit of using this is that a we have pretty good information about the sorts of people who use different social media sites it may even be part of the personas that your conversational design team has come up with and we're very likely to see examples from users on those sites in the data so let's look at a couple examples here uh so here we have uh some examples of someone on stack Overflow asking us chat bot to start playing music I would say these are not great examples of the intents but it's a good demonstration of the effect of prompting based on a specific site so my bot does not play music any suggestion how to play audio on another program any way to play YouTube music on Google home so these are questions you might see on stack overflow and some demographic things to consider uh like gender bias a little bit more likely to be male generally working age not a lot of children not a lot of very old people tends to be in more of a professional context and is of course for a specific domain um some other uh examples here so here we have Facebook um some examples of the output here are I need a mood lifter can you play something upbeat what's playing question mark what music are you playing question mark so um some qualities of Facebook it tends to be fairly gender balanced but it does skew older um I believe of all of the major social media sites Facebook has the oldest user base um you'll have a variety of contexts there's gonna be a big focus on news and current events and it's very broad domain and something I would point out in this data in particular is that the capitalization punctuation here are even more formal so following patterns are generally associated with with older language users on the other hand something like YouTube is going to skew quite a bit younger uh so here uh some of the generated examples here are uh how can I listen to your music I need to listen to some music uh what songs should I listen to so some general notes about using this uh type of approach to generate text Data the more unique and specific your intents are the less well this will work and that's because there's just less likely to be a lot of examples that are relevant in the training data so you are more likely to get unhelpful approaches you are more likely to get unhelpful data so something like asking whether or not this is a chat bot that the user is talking to you're probably likely to get pretty good results um something like asking for the specific version number of the chat bot you are you know perhaps not as likely to get as good results uh and the more you are trying to capture usage patterns that are well represented in the training data set so online the more likely you are to be successful uh and just a final reminder that adding data diversity in this way is really a stop Gap measure it's a way to give you a warmer start to get you going uh it's not actually representative of your users so your best bet is going to be once you have a system that works pretty well uh start folding in actual user data because that's always going to be the most relevant so if you've generated some data and you would like to do uh validation to make sure that you're not putting something in your system that will make it perform worse my general recommendation is to do hand validation uh look at the data that you are looking at you may even use you know an annotation tool like Prodigy to do a quick up down vote about whether or not this particular input fits the intended intent and if you would like to do a little bit more validation you could use an embedding visualizer like the one provided by cohere to make sure both that within you know different clusters you're having a mix of real and generated data ideally you don't want to generate very tight clusters of data that may skew your model output or you know skew your uh uh models ability to handle the wide diversity of ways that people Express themselves when talking to chat Bots uh and also if new clusters are introduced you want to make sure that you are happy with those results so here I have an example uh we have a little bit of a cluster down here at the bottom again this is from the cohere playground we have a point here I need a mood lifter can you play something upbeat uh which if you will remember is from the YouTube generated data however this data point quite near it says would you play some music please and this is from the human generated data from the slurp data set and the other cluster in the upper right hand corner is similarly a mix of human and generated data which is good that's what you want to see it means that you are not you're filling in the embedding space you are not creating new distinct clusters that have no overlap with your existing human data um so mix of original and generated data in this part of the visualization probably these points are fine to keep uh however we have another cluster over here with two data points one says what's playing and the other says what music are you playing uh and these are both generated data points and I would say that they also have a mismatch with the intended intent of playing music so this is more about asking for the track name so these I would either want to remove entirely or relabel as a different intent and if you kept them in they may make it harder for your model to correctly classify whether someone is asking to start playing music or asking about what is currently playing and you would expect different behaviors for those intents so as a general review large language models can help you with data documentation both adding volume and a little bit of diversity until you get real data they can give you that warm start to get your system a little bit more usable you want to prompt with as much as possible existing data or newly written data right use the human generated examples to Anchor and validate your your generated examples and then hand verify output just to make sure it is according to the standards of quality that you have for the project all right so I hope this is helpful and if you are working on building chat Bots and using large language models uh I hope that this helps you get started a little bit quicker uh and gives you some good ideas for prompting and ways to do um your work a little bit faster and easier [Music]