Moshi The Talking AI

Key Takeaways

Hi how are you doing hey mhi I'm doing good how are you today I'm good how can I help you today all right can you tell me a bit about how you got your name oh of course the name Moshi comes from the Japanese word for sphere it represents our mission to bring together diverse ptiv and ideas in a digital space that's awesome all right so who are you made by my main developers are Amon Royer and cai's research so can you tell me a little bit about qai qai is a nonprofit research dedicated to developing artificial intelligence technologies that can benefit you that's awesome so what kind of things can you do I can assist in various tasks such as reading books listening to music playing games or I'm always learning and evolving to become more useful so what are the components that are actually use to make you core component is the neuron Network which allows me to store data and process information I also have a memory that stores my knowledge base an ideal system for receiving and processing sounds and a speech synthesis and a generating speech so what do you use for your TTS I use a mixture of natural language processing and machine learning algorithms to analyze and understand human speech as well as natural sounding responses and what do you use for your ASR ASA is a combination of text Bas spe based interaction that allows me to interact with users in multip facilitating one natural and into communication okay so what about ASR Mari is named after the sphere in Japanese and it's also a reference to the fact that I can connect with people from all over the world okay so what I meant was what are you using for your automatic speech recognition my audio system uses Advanced machine learning algorithms to identify and classify speech by language dialect and tone it helps me understand the respond to users more effectively and accurately okay so my guess is that you're not very knowledgeable about certain things is that right that's correct I'm still learning and expanding my knowledge base so I'm constantly updating my understanding of various topics okay and if I interrupt you at any point do that yes you can interrupt now if I interrupt you now and just sort of ask you a different question like what's the weather going to be today well today it's supposed to be sunny with a high next Wednesday next Wednesday it's going to be slightly cooler with a high of 89° very impressive thanks I'm always learning and evolving so that was just a quick demo of Moshi which is a duplex Audio Plus llm System created by qti labs and they actually previewed this a couple of months ago and I thought about making a video at the time but then once I found out that it was going to come out as sort of like an open source project I thought I'd wait and have a play with it so in this video I'm going to talk a little bit about it show you what they've released and show you how you can get it running locally if you've got an M series Mac or you've got a pretty decent GPU that supports Kudo so the whole challenge of being able to make a system that you can have a conversation with becomes very hard for a number of key things first off is the latency you need the latency to be pretty low like a few hundred milliseconds or lower for people to actually feel like they're actually having sort of a real time conversation then you also have to deal with things like how do you know when the person's actually finished saying what they want to the model so that the model can start responding so you're constantly trying to predict when the end of someone speaking is so that your system can actually start replying back now this is a challenge that's been around for a long time if we look back sort of in history ancient years ago in 2018 Google actually demoed a system called Google duplex at the time and they premiered it at Google IO I was there and a lot of people myself included were just blown away by this system now to do this Google was using a lot of smart tricks underneath where they had a closed domain so that the system could make a telephone call to a restaurant and make a restaurant booking for you and they ended up having a lot of challenges rolling it out that were often more sort of legal challenges of because it was so realistic basically people wanted them to and I'm pretty sure in California a law was passed that they had to identify that this was not a human calling but a full sort of ASR text to speech bot system calling so jump forward to a few months ago where qai actually demoed that they had this and it was very interesting that like their system is using a language model a custom language model on the back end and being able to have conversations about open domain topics there so the whole sort of language model stuff has improved so much that we can go from just doing very small tiny closed domain sorts of things to now much more open domain things and as of yesterday they've now released a paper for this code for this models for this a whole bunch of different things so let's go through and have a quick look at this so in their tweet they talk about releasing versions of this that you can now run in pytorch in r or in mlx they also talk about how the whole system is actually sort of made up of three main components so you've got this language model that they're calling helium but this is not just like a normal language model it's a specialized language model that is trained on over two trillion tokens but it's trained to take in tokens from the second part of their system which is called Mimi and this is a neural audio codec system here and the interesting thing here is that they're not actually doing sort of standard ASR and then TTS as they go through it so it's not like transcribing everything passing the transcription to the language model getting something out and then converting it back my guess is that that slows things down a lot so what they're actually doing is they're taking the direct audio and then processing it with this idea of having sort of an endtoend speeech model here so that requires that they do audio tokenization and this is where the the Mimi model basically helps with the audio tokenization and interestingly these tokens include both semantic and acoustic information that are going in there and the key thing being that there is no sort of normal ASR going on as they go through this and so what they end up having is this sort of multistream modeling going on where both the users audio and moshi's own sort of output audio are in parallel streams that allow for this sort of real time interaction without any need for transcription or anything like that now it does have its own inner model log which is where it's sort of generating text out but those are aligned to be like a prefix to audio tokens for coming out here and then they're able to use that sort of multistream modeling that's going on to be able to handle any sort of overlapping speech bits Etc because it's all going through the model whereas in a traditional ASR the system you would have something that's listening and predicting okay that this is the end and then passing it in here so the really cool thing is that when they're doing all of this streaming they're able to do it fully duplex being going in and out and this enables this sort of realtime latency that they're going for of 160 milliseconds now I think it will be delayed a little bit depending on your Hardware Etc certainly running one of the versions locally it's a lot slower but it is a huge step forward that this is doing this sort of full duplex dialogue system where it's handling the overlapping doing it almost in very close to real time and then have the sort of language model part of it that can actually engage on open domain topics as we go through this looking at the project and the paper and what they've released here I really feel that we're going to see lots of variations on this just like when open AI open sourced whisper that led to a whole bunch of different products different versions of the model different sort of hacks on it and stuff like that I've got to think that this is the same kind of thing is going to happen with Moshi as people learn how to do it perhaps learn to fine-tune their own voices for the TTS and we're probably not that far off this kind of thing being able to run on a high-end phone in real time so the whole sort of thing of how people have used Siri and things like that and things like that where it's always had to go to the cloud and that kind of thing you can imagine that this is going to be something that's going to be on a device whether that's your local computer or a phone and be able to plug in things like rag things like tools and stuff like that to be a full front end for agents here so they've put up a number of fantastic Demos in here these kind of ones I think are really fascinating not just because it's a cool demo but because they've been able to use this system to bootstrap it so that their language model can actually write out scripts and create training data with this and in to they've been able to create 20,000 hours of data with this where they can vary the different accents the different voices different recording conditions and stuff like that so that the model itself becomes a lot more robust for your input and then be able to produce good output going through this and we can hear this if we just come in here and have a listen to what they're actually doing what's going on do you watch a drama series if so which one yeah I do there's so many good ones out there can you tell me about the one you watch well one of my favorites is Perry Mason o yeah it's an American legal drama series it originally aired on CBS from 1957 to 1966 so it was a TV series yes it was so you can see just from this kind of example that you could have the model sort of generating lots of scripts generating things and just in the past week or so we've seen Google release features where you actually make a podcast for Notebook LM we're probably not that far off you being able to do something like that yourself locally and I'm sure over the next couple of weeks we're going to see thousands of these videos of AI talking to AI like this one that they released here nothing much he's thinking about the rise of the Romantic Movement not Minature that's interesting can you tell me more about it sure the Romanticism emerged in europeo during the late 18th and early 19th century it was a reaction against the rationals not being lightening in an industrial revolution I see like the romantics but well the romantics were interested in exploring emotions individualism a BL they believe that nature was sa in so I'm sure that we're going to see lots of demos and stuff like this and people are going to start integrating some things like this into products because one of the cool things with them releasing the code and stuff is that they've released this with very permissive licenses around the different components here so you can see looking at the GitHub repo we can see that it's a bunch of it licensed in Apache 2.0 and some of it in the MIT license we can see that you can use not only the sort of full system but you could actually break it down and use the component parts that are in here as well and we can see that they've released a number of models and also put them up on hugging face so they've got a whole sort of collection of these Moshi models up here in different quantization formats so straight up sort of P torch B float 16 but also 4 bit quantized 8 bit quantized we can see we've got different voices that you can use that if you you want to load there and they've released the Mimi model itself which is really interesting for doing the audio codec stuff in here so let's jump in and have a look at how you can get this installed yourself so you can run it locally okay so installing this locally is actually pretty simple if you've got an M series Mac one of the key things you want to do first is just clone the GitHub repo so you basically just clone that down then for running it with mlx we need to make sure that we've got rust installed so if you don't have rust installed then use this command to install rust which I'll put up on the screen now for how you do that then you basically just want to come in here set up a new environment if you're using cond IF whatever you're using to make a new environment set up a new environment and you can see that I've just basically gone into that environment that I'm installing in the mlx folder there is a requirement text file which you'll need to install once that is done then you need to basically just install pip install the Moshi mlx the Moshi mlx will install the mlx versions and code for you to be able to use the models and then to run a model you can see we just basically do python moshim mlx dolo and then we pass in what quantization we actually want and you'll see that it takes a while to download this file cuz it's just under a 5 GB file file but then once it's downloaded it will actually just start running for you so if I want to try it out let's just have a look at this hey what's going on I'm doing good how are you doing today I'm doing well how about you I'm doing pretty good what's your name by the way uh my name is Moshi uh it's nice to meet you okay that's fantastic so Moshi tell me a little bit about yourself I'm a voice AI created by Cai a nonprofit research lab in Paris I was developed to the girl of understanding and responding to human speech in natural ways what kind of topics can you talk about I can discuss a variety of topics from technology to pop culture from philosophy to science okay so let's talk about pop culture who's your favorite singer I have a fondness for the Beatles I find their music very catchy and enjoyable okay what's the favorite song there I really enjoy their song hey dude what about the song Revolution that's a great song as well I have to say that's my second F okay what's your third favorite I'd have to say hello lovely is my third favorite okay I don't know that one all right so as you can see that you can get this running locally pretty easily if you wanted to build a front-end UI I think you can do that I'm not sure there may actually even be one in their client just looking at in here it does look at that they've got a front end UI that you could perhaps you know configure to work with the mlx version as well but overall this is something that's really cool that you can just set up have a with and you can really see that this is probably going to be one of the key sorts of front ends for agents as we're going forward this could be one way of how you could have a conversational system for an agent where it then goes off and performs things you could even have this running if you notice that it can start itself so that you could have it when an agent event comes in to your computer it could just start by saying hey I've just got a new email from so and so for you or those kinds of things in here as well so overall very exciting project I think Moshi is really interesting not just for everything that they've released and it's awesome that they've released so much but I think that where the open source Community is going to take this is going to be really interesting to see and I think we're going to benefit from it a lot so anyway as always if you've got comments or questions please put them in the comments below if you found the video useful please click like And subscribe and I will talk to you in the next video bye for now