Chat with Audio Speech using LeMUR - AssemblyAI's LLM Framework

Key Takeaways

imagine you have a 10 hours of audio clip and you want to use that audio clip to ask questions using a large language model that is currently not possible easily like you have to set up your own Vector database you have to do a bunch of the things but assembly AI has come up with a new model called Limer it's actually a framework that can apply powerful large language models to transcribed speech so with lemur it's completely possible for you to transcribe a 10 hours audio clip into an audio transcript and use large language models for that like it's about like 150 000 token and you can ask questions or do summarizations and many more tasks and in this video I would like to dive deep into Limer and we're going to see also an example lemur as a model has not been released it's you have to join a waiting list but there is a playground where you can play with it's quite incredibly amazing and I would like to show you what I did and then how do I feel about limber to start with water Slimmer it's a new framework for applying large language models to transcript speech with just a single line of code lemur can quickly process audio transcripts of about 10 hours worth of audio content which effectively translates into approximately 150 000 tokens for tasks like summarization and question and answer now if you are familiar with the large language model space you know hundred and fifty thousand tokens as a context window do not exist like in fact yesterday we covered a video where we did hundred thousand tokens and we celebrated like anything so what is this what is happening behind the sceners without limit you can probably approximately process 45 minutes of audio using a large language model because of the context Window 8 000 tokens I mean forget about anthropic Claude 100K here just default 8000 token but with limit what you can do is now you can do 10 hours of audio content which is approximately 150 000 tokens and how is it happening it's happening because there is no black magic behind it basically what open a sorry assembly a has done is they've created an intelligent system like they've created a turnkey solution that has got a pipeline of a vector database like Chain of Thought prompting and self evaluation as you can see this is the architecture of limit so lemur is at the center and you can give the transcription first the audio goes inside they've got a model called conformer conformer does the transcription then audio goes inside the transcript ID and the text data so now that goes inside a transcript database as you can see it doesn't directly go inside a large language model rather there is a transcript database where this is going inside and now that has been segmented intelligently into multiple segments lemur segments the transcripts intelligently like we don't know what that intelligently is but as you can see it gets segmented into multiple like files or indexes or segments that goes inside a vector database now this individual components go inside large language models those are finally stitched together using a Chain of Thought reasoning and self-evaluation and all these kind of techniques and now finally that goes inside another llm and then the response comes back to the user as a limit response this is like honestly speaking this is an incredibly intelligent architecture because now what you can do is in just a single call just a single APA call you can send one big audio file and you can get the response whatever you want to do back it uh to be honest it unlocks a lot of possibilities that were not possible and one of the possibilities that I would like to show right now like right now which is again to show that this is something that I tested and it is amazing I was watching this interview it's an interview by a stripe founder Patrick Collison with Sam Altman I really enjoyed the interview but um you know due to work I could not finish it like it's a it's a 52 minute interview uh I kicked about like um 24 minutes and I thought okay what if I can give this to lemur and then ask questions to a level so that's exactly what I did I copied the YouTube url I went to the Lemur playground as you can see here and inside lemur playground I asked to transcribe it for me and it did as you can see the like the entire transcript is available here until the end and I can also listen to it I mean it's not like I cannot I can completely listen and Patrick Carlson all that again so 54 minutes like close to one hour of audio clip is available got transcribed in less than 10 minutes and what I can do here is now I can ask question so all these things like whatever is here it goes inside this large language model I mean I mean unfortunately I'm not using like a 10 hour audio clip probably I need to take Alex Friedman podcast or Joe Rogan podcast here but let's say I'm just taking this and I've given it to limber and I can ask questions I can ask questions like summarize this transcript like I'm 12 years old I want to ask specific questions about the transcript how could I improve this transcript for more engagement nothing specific I just want to play with limit features I'm going to say I want to ask specific questions about the transcript so what is the question that I want to ask I know at some point in this interview there was a discussion about nuclear resources okay so I'm going to say what did Sam Altman say about nuclear resources and it's comparison with AI okay I don't know if it's going to answer this is the first time I'm trying it out so answer format and that's completely fine so I just said what did Sam almonds say about nuclear resources its comparison with AI and I know for sure like I listened to it I know what he said so I I can see very well if it hallucinates or if it doesn't hallucinate Sam Alban said access to nuclear resources more difficult than for AI due to complex permitting process um this is like one of the one of the point here but you can actually see nuclear you can see here first of all he was not very interested in people comparing these things and there were a lot of other um um secret like um a lot of other questions about nuclear secrets even let's see when I specifically say nuclear secrets instead of nuclear resources what it says so what did Sam Altman say about nuclear secrets and its comparison with EI and uh I'm not editing this video it's live as you can see let's say that while classifying nuclear secrets is probably a good idea it's not a complete solution on its own he thinks the biggest lesson from nuclear non-proliferation for an AI safety is creating an international regulatory Agency for powerful AI system similar to iaea for nuclear technology this is super impressive I am telling you you can listen to this podcast I definitely listen to this piece and this is like this is the context of what he actually said and we can ask a summary also like I can say uh do do a summary or I can say AI coach how can I improve this engagement I don't want to do coach let's say I want to do a summary and describe the context and optional so again just click generate and this can create a summary for me so now what we have seen like like let me let me take you back to the start we have been introduced to a new model from assembly AI which is not going to be a free model to be honest um because this this is not an open source solution so lemur is a solution called leveraging large language models to understand recognize speech so with one APA call you can transcribe an audio clip or YouTube clip or whatever it is that's up to 10 hours of content which is approximately 150 000 tokens and then you can start asking questions thanks to their very intelligent architecture I'm still amazed by their architecture so the summary is this Sam Altman and and an interviewer oh it's Patrick Collison AI discuss aai Innovation and they're talking about what um they did a accelerated science scientist Capital few talented young Founders exist today something wrong as there were many like Elon Musk before the key points are open AI secret advances AI assisting replacing scientists uh lack of top young Founders so honestly I wouldn't say this is like a great summary because from the 25 minutes I listened it's not like the perfect summary that I would have probably expected but I don't know what happened in the remaining 25 minutes as well like so could could be like the entire conversation went in this direction it is super amazing and um you can see that it unlocks a lot of potential imagine like you're a student and one day you could not go to class and um you have got your students have got recordings like maximum what you would write in eight classes it's like eight hours or six hours now you can plug in all the eight or six hours inside this and then you can ask questions education could be completely transformed with this thing so what lemur unlocks is you can apply llms to multiple audio transcripts you get like safe output um in customer support you can inject context-specific use cases like for example you can say did the sales representative follow the exact procedure process of prospecting and you can supply a sales Playbook so now an agent can be evaluated based on the Playbook rather than just generally knowing and I I wouldn't probably like this kind of evaluation but you know this this part of life it's modular you can do first integration under the state of the art so what assembly a has also let you do is you can just click this playground and start playing with stuff like there's a three hour customer call MP3 there is a poetry super power this is a popular Ted Talk um it's a talk about you know dyslexia and all these kind of things and you can also Supply a bunch of things like for example there is a Django course Django course Django course Django course and you can say oh what can the instructors do better okay it looks like it's something related to my profession of being a YouTuber so let's go to the go to the playground and then say um how could I improve the explanation of Django so that High School students can understand okay so basically let's say I teach Django and if I'm somebody who teaches Django oh this doesn't look like Django stuff this looks this looks like some supreme court stuff okay cool maybe I came with a wrong transcript that's fine maybe it's going to probably most likely hallucinate and then give us something okay this it has given me answers but as you can see um it is pretty good and limos that stands for leveraging large language models to understand recognize speech is currently available as a waiting list so you can join the waiting list and once you join the waiting list you get access Early Access to limit as a product you can play with the APA but let's say you do not have access just like me all you can do is click the try and playground and go there and then you can paste a YouTube link or upload an audio and then start playing with the Lemur incredible piece of technology but I think this is also going to give idea to a lot of startups about how to build this kind of solution where you can now artificially increasing the context window of a large language model using some intelligent TurnKey solution quite exciting times let me know in the comments what do you feel about it see you in another video