P.1 Chatbot with Mic input/Speaker output using Python, Jarvis, and DialoGPT

Key Takeaways

So, the response time here is about 3 seconds or so, and I want to speed that up. Thank you. [Laughter] What's going on everybody and welcome to a video where yet again it will be a Syntax Snickers video uh playing with when I think I can be uh at least at the very basic level a endtoend chatbot with audio input from you the user and then audio output from the machine. Now there are many things we could change along the way. Uh and the kind of heart of the chatbot would be whatever the you know natural language processing chatbot um logic is right uh and I am working on that of on my own but I also stumbled upon this dialog GPT so it's this dialoguebased GPT model uh that is based on Reddit I will admit I have I've seen a little bit from it I don't think it's I think it's been cleaned up. It seems to be a very um well- behaved bot. Uh so I'm not I I I wish it was a little more spicy, but uh it's a very good chatbot. Um so I think that's what we're going to use at the heart of the bot. And then now it's just a question of stringing together um the speech to text. then run the speechto text through dialog gpt and then uh text to speech back out and boom you have a live uh chatbot. So how cool is that? I just want to see um I want to experience the kind of like live chatbot you know via audio. Um this is not something I've ever had besides maybe Siri or your, you know, Amazon Alexa, that kind of stuff. Um I've never been able to do it myself. So, um, so yeah, let's, uh, let's go ahead and get started. Again, uh, kind of like the other tinkering episode, this is going to be mainly a lot of kind of, um, um, hacking around um, and and just like copy and pasting chunks of code from various places uh, to get this just to work. And then from there, we can kind of fine-tune it and do um, arguably all kinds of really cool stuff. So, um maybe more to the series uh will come, but to start, we want to get this Dialo uh GPT and check it out. So, first order of business is this is located on HuggingFace. Now, I'm sure many of you guys are familiar with HuggingFace. Um I think HuggingFace got real popular while Daniel and I were writing neural networks from scratch, #nnfS.io, by the way. Um, and so I just kind of I didn't have time to really look into it. And then also, no offense to anyone at HuggingFace, but I hate your name. I just I hate the name Hugging Face. And so I've just always kind of ignored this website. Every time it comes up, I'm just like, um, but it's it's it's a very good a very good website. And I'm sure there are plenty of people who absolutely hate the name Sentex as well. So, you know, whatever. So, so, um, anyway, we're going to get Dialogo GPT from hugging face. And from what I can tell and understand, hugging face, I believe it's purely transformer models. Um, a lot of like language models, stuff like that. And the models you are like hosted on hugging face and then you use the transformers package, uh, which is, I believe, hugging faces package. I hate that name. Um, anyway. Um, I think right. I'm no expert on HuggingFace, that's for sure. Um, but I will say despite the silly name, uh, it it seems like a pretty cool website. So, we're going to grab, uh, this Dog GPT. So, it's actually super simple. You're just going to pip install Transformers. Um, and then this is the one that we want. Uh, and then if I scroll down somewhere, right, this is all the code. like once you have the model you just run the model with this code and again that's why golly it's so simple so um and then obviously you would need pietorch um again this these these tinkering episodes I am not doing everything and showing you how to install torch all that stuff by now there are a million tutorials so if you need help installing or doing pip install transformers um I you can you're going to have to find that so so anyway it's as simple as taking that code. And so we'll come over here and I am going to touch um one uh we'll call it getting started.py. I'm just making this file for us. Uh and then I can open that up. Let me pull this over here. And paste in that code. Now I've actually already downloaded this, so it should hopefully fingers crossed work out of the gate. Now of course that never actually works. So, let's see what actually happens. Um, I'm just trying to make sure I get this nice and full screen. Ah, perfect. And then let's go ahead and make the text bigger so everybody can see it. Okay. So, uh, we'll go ahead and save that. And in fact, um, what did I do? What did I do with the, uh, directory there? There it is. I'm going to pull it down here. Um, I think we'll run it from a terminal just to be sure everything is working as expected. So, uh, let's do Python 3 one getting started. Again, uh, if this is your first time running it, chances are it is going to, um, oh, a couple things. First off, let me also watch Nvidia SMI. Let me pull this over. I want to say I'm guessing it's not going to run immediately on the uh on the GPU. That's usually how like models don't usually by default run on the Okay, so we got the thing coming up. So, uh, let me make the text a little bigger. Hello, General Kenobi. Better be the response. Dang it. Oh, you know what? I think it's supposed to be hello there. Let's see if we can get it. Whatever. Um, let's see. Hello there. Hey there. Um, okay. How How are you? So, yeah, response time here is about, I don't know, 3 seconds. I'm I'm just kind of counting in my head. Um, I'm great. How are you? So, typing just sucks, right? It'd be so much nicer if we could just speak to um over the mic. So, anyway, this is dialogue GPT and again, it's been trained this has been trained on uh Reddit conversations. So, like trees of Reddit conversations. So, if you do something like um uh do you like Star Wars? I do. Oh, shoot. We Okay, we'll restart it real quick. I just want to show you that it it does know history. Of course, since this is going to be a live demo, it's probably going to screw up. Um, and then while we wait on that, um, I think I'm trying to decide like how I want to, uh, we might just leave it on the CPU for now, but later you can, it's something you Well, shoot. Now that I think about it, I think this is using torch. Hold on. Uh, do you like Star Wars? I like Star Wars. What do you like about it? So hopefully it'll be something Star Wars related. I like the characters. So at least it knows, okay, it's a movie, it's characters, it's a story characters and so on. Um, in the in the past I asked these the same like series of questions and it said it liked the lasers. So it just depends on what's what's the history. So uh, for now we'll break out of this. So that's that's the internal bits of the chatbot. You can you can tinker with it more if if you'd like. It's pretty cool to be honest. Um, and unbelievably simple to grab this. I mean, come on. Um, it's like I'm just I'm baffled at how simple um using that silly named website is, but cool. So now what we want to do is we want to take this as the internal guts and we want to wrap um around it h taking microphone as input and then um and the way that and transcribing, right? So, the microphone is input has to transcribe that to text, we're going to use Jarvis. I think we've done a few videos on that by now. So, you've seen that work. Uh, so we'll do that. And then, um, once we've translated that to text, feed the text through this chatbot. And then once that's done, just use text to speech. Again, uh, I'll probably use Jarvis. I have poked around a little bit for other text to speech because I'd like to have more voices. As far as I know, the only Jarvis voice is like that stereotypical kind of um LJ speech data set voice. And it'd be fun to have other voices. So, um I am kind of poking around. Mozilla has like the craziest looking texttospech library I think I've ever seen. Um it just looks like quite a bit to learn. So, we'll see. Um but anyways, so yeah, that this is pretty cool. We're off to a good start. And I think um I think at least what I'd like to do is take microphone input and get to um a response from the microphone input. So uh coming over uh yeah here I thought I had a directory somewhere with the Yes. Okay, cool. So, uh, let's just do original. Um, uh, maybe I'll just copy them in here. Yeah, I think I'll just copy them. No, I want to keep it organized. Original Jarvis files. And then, uh, let's just copy those. Boom. Okay. So, we've got a few things here. The first thing that we're going to do is the transcribe mic. So, um, there's kind of two things that you can do. You can do like this offline transcription and then you can do like from a file and then you could do transcription live from the microphone. And um I I like the live one. Um so this is the one that we're going to use. And quite literally I think all we have to do again like we've done before it's just merging these two things together. So rather than text input so we come down. Where is our text input anyway? It'll be some sort of input. Oh no. Oh my gosh, my brain is already failing me. We've we've we've not even gotten started. Um, this would take the input. This is transcribe mic. So, it's just whenever the mic comes in and I would propose the point at which we need to transcribe is going to be in this listen print loop. And then if not result is final, it just does all this. But then here, this is where uh we will once it is the final um that's what we want to pass in. And I think that will work good enough cuz I it seems to know like it seems to know when um and in fact, let's just do it really quick. It seems to know when you're like done talking. I don't know what they're doing in the back end, but it seems to understand as you're talking to just like keep trying to generate the audio. And then when you're done, it's like, okay, done. And like if there's a a long enough space of like dead um dead noise, I don't know. I can't think of a good word for that. Um anyway, uh so let's just do transcribe. Oh, wait. Python 3. And let's zoom in a little bit so people can see. Uh transcribe mic and then server DGX a 100. Oh my gosh, I think that's everything we need. Wow, that is so fast. Okay. So, then I'm going to wait for a second and then it's done. And you can see like it just finished and took us to a new line and then it does this spam thing that it does sometimes. Okay. So, it's at that point that's when like when it took us to that new line, I believe that's this else statement right here. Um, so that is where we take our transcription and then we feed it through the chatbot and hopefully we get an answer. So, um, arguably we just copy and paste this code into here. And here's how I'm going to do it. I'm just going to make a ton of space real quick. Come over here. Control A. Thanks. Uh, controll V here. Uh, we'll fix this. Paste up that. And then here um okay so this one will be somewhat challengingish. Uh so what happens here to understand like one we could just toss history. We don't even have to use the history. But the history is kind of cool that it like remembers what you've been talking about. So you wouldn't want to get rid of that uh cuz that's a really cool aspect. But you could and you could just reset it every time. But essentially the history is controlled by this line right here. Right? So it's tracking the step that we're on. So if we're on step zero, it's basically going to tokenize the input that like as we speak, it's going to or as we input text in this case, uh it's going to tokenize that text and either it's going to input um the the new tokens, right? Because it's just going to concatenate here um these two things together, or if it doesn't have a history, right? they're just assume, you know, they're using the step number to just know is there a history? Uh, and if there's not a history, then it's just boom, we'll use new user IDs. I kind of think you I think the smarter way about this would have been to maybe make chat history IDs initially empty and then you would just keep cat. I'm sure there's a reason. I don't think the people at Microsoft don't know what they're doing. So, I don't know. Um, it just seems like that would be better. Then you wouldn't you wouldn't need to be tracking step here, right? it would save you a whole line. Well, not a whole line, but it would save you the if statement, I suppose. Anyway, not important. So, what we're going to do is um I don't know how we're going to retain that history, but we'll we'll we'll figure it out as we go. So, first I'm going to cut this. We're going to go up here and past that and then come down here. And essentially uh for step and range, so this like loop that's going on um I I think we could cut that and and then we'll just come up here and paste that there. I don't know if you guys have noticed, but I really I I enjoy uh showing everyone that like this is legit like what I spend a lot of my time doing is just trying to figure out making stuff work this way and it's a lot of fun. Where's this code? Okay. So, we're going to get rid of that. Uh, probably should have saved that, but I like in some sort of file so we could reference it again later just in case we need it. But I think we'll be okay. So, uh Oh, did we What did we do here? Did we I think we replaced too much on that. I was too busy yiking. I think we Yeah. Why' you do that? Uh, we want it here. Okay. So break this over. Okay. So then so this is this. Okay. So all that's getting input into um into this chatbot is just whatever the user input via text. Well, rather than inputting via text, we want the transcript press plus overwrite character. So, we'll just print um or we'll say mic in equals that pasta. And in fact, where's our uh Yeah. H kind of want to bring in my curt my uh my curses or curtsies uh friend. This is my new favorite import for everything. So, blah. So curtsies format functions red bold green on blue yellow cyan and red blue. Okay. Oops. So let's come down here and just to make it clear print. Um so I'm going to actually do this print string green mic in. That's probably already a string, but just in case it's not. Boom. Um, like I don't understand like what's overwrite characters. So I'm kind I don't know. I'm not going to sit here long enough to figure out what that is, but that doesn't make sense to me. So, okay, whatever. Um, okay. So, that'll print out in green what we're going to put into the mic. And then basically, and in fact, let's do this. Let's make this make sure that is a string. I'm sure it is, but just to be safe. Um green mike in and then so Mike in then comes here there. Okay. So then uh what's going on here just for the record is um it's essentially encoding whatever you input the text that you input and then it adds this end of sentence token uh and then it um it's just tokenizing that if you have a history it's going to essentially append to that history uh with this line. uh if you don't have a history then what you input is just what you just you know input here and then you feed it through the model it generates a response and that response is in theory like it's all it's tokenized or it's encoded into these like um numerical label not labels I don't know it's tokenized okay numerical ids is probably the best way to call it um and then at the very end when you want to see it in actual real language it gets decoded and then you're just decoding the output um up to probably the you know it's just going to keep generating until it reaches this like end of sentence token that the model learns to output and say hey that we're at the end now. So if you were to sit uh if we set this to false you will probably see literally end of sentence uh like this little token for the end of the sentence but we'll leave that for now. I don't think I'm going to change that. Um okay so we've made our changes. Let's see if if we're done. Um, mic chatbot. Um, oh, wait. We have this one here. Yeah. So, oh, we're going to need the server. Do we have that? Okay, we do. I think we're good to go. I don't know. Uh, Python 3 1 getting started. And then we just need to specify the server is the DGX A100. OMG. No. 254. Oh, did we leave that little snip? We did. Um, and then listen print loop. Where does that get called again? Here. With microphone listen. Okay, I think that'll be good enough. Uh, where is where's my console? Cool. Let's try again. Hello. How come the other one started the other one when I did the mic? Oh, is this maybe loading the model? That's probably what takes a moment. Oh my gosh. Uh, didn't we bring that in? Oh god. Uh, I knew I knew I hit my middle mouse button. It was this. Yeah. SRV. I knew I hit my middle mouse button and uh I looked to see like, oh, did I paste something? I didn't see it. But anyway, that was just a stupid I typoed basically. Testing. We don't have the step. Of course. Uh, I will do you guys a solid and I hopefully I'll remember to edit out these weights. Um, okay. So, step, how do we want to do that? I I'm pretty sure the loop is is always inside of this function. So for oh gosh for response in responses then it goes here man I can't decide how I want to handle for that. So I think at the at the very be basic level I think we can get away with just passing here step equals zero and we'll we I think that will be okay. Okay. So, let's let's just keep taking swings until we get a hit. Okay. Here is another test. Hello. How are you doing today? We did it. Doesn't that look cool? So, um I guess are we waiting? I'm not sure if Oh. H. Okay. So, that's pretty cool. Um, I'm going to go ahead and break it here. That's fun, isn't that? I just I don't know. That just gets me all giddy inside. And how easy was that? We did this in what? 23 minutes. And I probably could do it faster. Um, and arguably, I mean, that was what, like a couple of moving some code like three times, um, making a quick fix and then boom, done. Um, I'm guarantee you there will be lots of little issues with this as time goes on, but what a great start. What a great start. H, I get that just makes me so happy. It's so cool to like talk into the microphone and then get the response. And then now what we have to do, and I think I'll do it probably in the next video, is um um we need to do audio output, right? And then once we have audio output, uh it's like a full we've done it. Like it's a it's just this full end toend uh chatbot that you can like literally talk to. And uh this was quite fast. Um, where do I h that is one thing I wouldn't mind uh possibly fixing. I'll remove the wait time here hopefully. Harrison in editing don't forget remove that. Hey, what's going on? How are you doing? I'm doing well. Thank you for asking. So, the response time here is about 3 seconds or so. And I want to speed that up. Thank you. [Laughter] Okay, cool. Um, yeah, let's go ahead and uh let's fix that real quick. Uh, and the fix to that is actually super simpleish. Um, I I'm trying to decide what I want to do. I think I'm going to copy pasta. Uh, where's rename? Am I blind? I guess I misclicked it. I Okay. Uh Dialog GPT testing. Um I think what we want to do is Where is my cool? Um take this. So copy that begon pasta. Got a little too excited with our copy. That's okay. Uh, and with Torch, it's like this like, um, it's one of the things I don't like about Torch. Whereas like with with TensorFlow, you literally just do like with TF. And then it's like, uh, what is it? SlashGPU, and then you pick a GPU, and then, you know, colon, and then bada boom, bada bing, you're done. Not so with PyTorch. Um, it's a little more annoying with PyTorch. Let me see if I can find the uh, it's like you got to do like dot and it's like to device and then you have to pass like a device object maybe. I don't know. I got to I got to figure this one out. Um, let's see. Uh, PyTorch device choice. I don't know. PieTorch. Uh, how about PyTorch run on GPU? Where's the code, bro? Two. Oh, interesting. So, can I just say CUDA? Is that going to work? Uh, okay. So, I think what we would need to do is send um we definitely want the model tok the tokenizer probably doesn't need to be, but the model needs to go two. Can I really just say CUDA? I don't think that's gonna work. But if it does, cool. Uh, tokenizer.ccode. I don't think I want to encode, but I do want the model generate and then the tokenizer decode. I think the tokenizer can stay CPU. That's like a quick lookup and it's only like I think a uh vocab like 50,000 or something like that. So I don't that should not be slow. What should be slow is probably the model. So, um, Python 3. What is this? Dialogue. Okay. Okay. So, it's going to take a minute for this to load, but then once it loads, and then also watch Nvidia SMI, we want to make sure that one of these GPUs actually loads up, maybe two gigs, five gigs. Okay, looks like the model's about well, we were already at a gig, so um maybe 4 gig model. Uh and then if we say, "Hey, what's up? You little what? Two cuda model generate.2CUDA. What's your problem, bro? Uh, okay. Let me think about this for a second. So, two CUDA. So, that definitely worked. It loaded up our model. I think what it's getting angry about is this. These user inputs are not on the model. So, I'm going to say two CUDA there. The concat operation. I don't think that would require a two CUDA, but then model generate would need a two CUDA, which we already have. I don't think we're going to need a tuna on that. Let's go ahead and test that though and see what happens. Hey, what's up? Not much. You right. Okay. So, as you can see, this response is not 3 seconds anymore. This response is like bang, done. So, um, uh, that's a bummer. Like, watch how fast. Enter. Boom. Done. Oh, here's a good that's a good example, too, of the history, right? Like, cold and raining. That's a bummer. Yeah, it's been raining for a few days now. Um, yeah, that's a that is so cool. And we literally just grabbed that in like one second. I just think it's so hilarious how quickly I have I if only you knew how much I've struggled with making like a Reddit based chatbot and then I grabbed this in like two seconds. It's It's funny to me. Anyway, uh okay, so that's way quicker. So now last thing before I end out the video is let's port over the moving to CUDA. So that's not too hard, right, to docudd. again. Gosh, it's just so much easier to just like wrap everything with the TF device. Um, and that's only if you need to use a specific device. Like it's going to automatically go to your GPU. You don't have to say like each each individual line where you got to do two. Um, okay. So, I'm going to get over here. I'm going to just do this. I know you guys can't see the one that's on the left, but I'm just going I'm just porting over the changes that we just made uh as soon as I find it. Um wrong file. Isn't it fun? It's so fun. Uh okay, so two and then here will be qua cuda and then we'll come down here. We are almost done. Uh this is to uh CUDA and then a little late. Should have done this a long time ago. Copy. Uh the cat we don't need. Paste it there. And I think that's it. And boom. Now we should have a really fast How dare you send me a text. Uh we should have a very fast uh model. So let's go ahead and test that. I'm going to open yet another terminal because why not? It'll be easier than finding the one I was working in. Uh, so Python 3 one getting started and then we need the server DGX A100. Oh my god. And then we'll get things started and then we'll be done after this. Hey there. How you doing now? Um, I kind of want to do one more thing and change his color to like orange or blue or something. He thinks it should be red. So, we'll we'll make him red or her. Oh, this chatpot just I It gets me. I just It gets me. Um, okay. Uh, how do we do this? Format. I don't think we can wrap that in red. Um, GPT. Um, okay. So, one more little tinkering and then really we're done. We're done after this. We still have to do the audio out, but anyway, it it's just straight up from here. It's just like text to speech. Boom. Done. I mean, that's it's just so cool like how quickly you can dev this stuff. Uh, so the output uh print tokenizer decode. So really we would just say output equals and then this I would probably clean this up further but that's okay. Uh red output um not going to be heartbroken over an extra space there. Might actually be useful. So why is it still at five gigs? Didn't we break it? Although I have a ton of windows open. Maybe I left one open. I have no idea. Okay. Uh let's just rerun this. That's kind of weird though. Why is that still five? I swear I closed them, but whatever. Hey, what's going on? Do you like your new colors? Okay, I get it. You like the new colors. Very cool. What's wrong with the logo? Okay, maybe we'll make it a little smaller. Okay, you get the idea. Pretty cool. Um, now all we're gonna do Thanks. I'll try. Uh, I'm gonna try to do. Okay. It's so fast at response. I just think that's so cool like how quickly we can get a response. And a lot of them have been pretty good responses, too. Same. Same. Okay, that's all for now. Uh, in the next video we'll actually give it a voice. Uh, if you have questions, comments, concerns, whatever, feel free to leave them below. Otherwise, I will see you guys in the next video.