ElevenLabs Conversational AI Webinar

Key Takeaways

so welcome everyone to today's webinar on 11 laabs conversational AI platform so we're thrilled to give you a sneak peek and have you join us as we dive into this exciting and transformative technology now conversational AI revolutionizing how businesses engage with their customers uh delivering smarter faster and more personalized interactions in this session we'll explore how 11lbs conversational AI platform takes innovation to the new level generating intelligent um apologies from seamlessly converting uh Speech to Text generating intelligent responses and delivering premium qu voice quality outputs this platform streamlines workflows enhances user engagement and empowers teams to create memorable customer experiences now our goal today is to equip you with actionable insights real world use cases and a clear understanding of how 11 Labs can help redefine your approach to customer engagement operational efficiency and AI powered communication so with that in mind let's get started so today's session will be led by me alongside two of our top Engineers who have been deeply involved in developing the product will'll be showcasing today hment and yza in terms of the agenda we'll do a very brief introduction to conversational AI at a high level and then we'll be able to see um the conversational AI platform in action as demoed by hikmet and finally uh we'll do some practical application or rather we'll showcase some of the Practical applications of the convo AI platform um before ending with a Q&A session now some housekeeping before we get started so this webinar will run for approximately 45 minutes during that time feel free to drop any questions that you might have in the questions uh section uh you can find it in the bottom right hand corner so we'll be answering some of them during the session itself and some of them during the Q&A part of the session we'll also periodically drop uh some poll questions so please uh to respond to them and this webinar is being recorded and a link to the recording will be shared uh with you after the session and any technical issues that you might have feel free to reach out in the chat so 11lbs uh conversational AI is a platform for deploying customized conversational voice agents built in response to our customers needs our platform eliminates months of development time typically spent on building conversation Stacks from scratch now the platform combines three key components the speech to uh to text portion the um language models and finally the text to speech using more human analogy the ears the brain and the mouth now with the platform you'll be able to get production to production apologies in a matter of hours instead of months and bring your own business logic to the mix should you choose to do so and also get realtime analytics from the conversations uh your users or customers are having now with that brief Overture I leave the floor to hickmet and ysf who will be showing us what that looks like in practice take it away guys perfect yeah thanks a lot bdan uh so yeah I will just uh I will just do a a very quick intro on our on basically the platform that we have built and just like highlight a few key kind of considerations and key points of of conversation I have from 11 Labs uh where which we try to kind of like focus on while developing the the product over last um almost almost a year uh and then yeah Hickman will go go ahead and show the demo so basically when you when you go on our landing page you can kind of like he he kind of like all the points are kind of mostly highlighted but I I just I just uh say a little bit more so what basically conversation in a nutshell is like combining the three crucial steps which is speech to text then the brain or agent which is the llm which can have have some kind of function calling some kind of knowledge base and um and so on which can be also like either custom one that you can plug in or it can be it can be Gemini it can be Claud it can be whatever you want and then the text to speech which 11 Labs is is very famous for into a single single product and here are kind of like a nine things that are kind of like highlights of of of what we try to achieve so one thing is like low latency uh because we are kind of vertically integrated we self host the the transcription service so from speech to text we self host the text to speech part we self host the turn taking the orchestration the interruption and the voice activity and we only make a single call for the llm uh we are vertically integrated and that's why we can kind of achieve very low latency also in our implementation we will really try to focus on having very high quality at the same time it's not like low latency at all costs like for example when we know that the user let's say is thinking we actually try to give uh give time and this will also like improve as as this product will mature give the user time to think and not really like jump into uh their thoughts and that's that's basically this this Advanced turnt so as we have a strong research background we try to solve this in a more researchy way to really provide as much as humanlike conversations with AI agents then you can bring any llm so yeah just as I mentioned there's a lot of flexibility you okay you can bring your custom open AI API protocol support the LM as well we support external function calling and like stuff stuff around that as for example we try to try to optimize the experience to be to be natural even when let's say there is a there's a longer longer timeout for for a for for a function call we support all the Library voices or like most of the library voices that are on the 11 laps platform you can directly use them in the convey agents and you can also fully audit audit any costs that that you made so for example if some if some user give you a gives you a thumbs down button for thumbs down for some response you can see in the in the transcription at which point during the audio the user didn't like the response and then you can you know investigate why that happens conve is also fully multi multilingual so we support 31 languages based on V2 or v2.5 uh uh text to speech model and uh the latest latest upgrade is the native to integration so you can bring in your your trilo numbers which will be natively integrated into the conveyor platform to to to handle your incoming uh calls so that's kind of a brief overview if you would have more like technical question questions of you know what we are trying to to achieve here uh uh going forward we can we can answer them at the end and now I will give back the the stage to hikmet who will actually show you show you the platform thank you very much Joseph and just to see what we mention everything in in real life I will share you today with uh two demos one of them will be more focused and very similar to what we actually deployed on our documentation page H Live support agents for the 11 labs um in this case uh just to go over the platform very quickly you are able to Define your agent language in here we we support SE one languages and for the first message this is the message what uh the agent will will say when it opens the platform and then you define your system prompt and in here we also pass the knowledge base basically some knowledge based from our documentation these are some information around the conversation AI you are able to add multiple different base knowledge base whether you can drug and drop a file in multiple different formats or you can just pass a URL to your web page or your H desk or wherever you want to retrieve information and we will scrape it or you can just copy paste your knowledge in here in here we provided some both uh documents copy paste and URLs as well and it's extract information for us and let's go and actually test our AI agents in here hi this is Eric from 11 lab support how can I help you today hey Eric can you give me some information around 11 lab's products certainly 11 Labs offers a range of products centered around voice technology our core offerings include text to speech which transforms text into realistic speech and speech to speech which changes a source voice to another while maintaining its original characteristics me a bit more about the conversation I heard it's the new product 11 Labs conversational AI is a platform designed to deploy customized conversational voice agents quickly it combines three key components converting spoken language into text processing and understanding context to generate intelligent responses and transforming text responses into natural sounding speech this platform is ideal for creating what's your pricing model or like how much does it cost the pricing for 11 Labs conversational AI which is currently in beta involves credits for usage during the beta period setup and prompt testing cost 500 credits per minute while production usage is 1,000 credits per minute okay thank you much ER that's all for today you're welcome if you have any more questions in the future feel free to reach out have a great day as you can see like these are the some information we provided from our 11 Labs page I can also quickly show it these are the information where it was able to basically like provide the usage pricing page and we also provided the product page where it was able to pull the text speech speech speech API and Etc and it was able to understand this information and use it when it's answering for us um and if we go back we also added some evaluation and data collection basically you can use you can use these evaluation criterias to understand how each conversation is went so for example we were checking whether the conversation was positive or negative did we solve the user inquiry and we also asked to make some data collection related with what was the issue type and what was the product category where it was asking and if you go to conversation actually we are able to see here first of all the entire conversation itself you can replay or download audio you will be able to see the summary of the conversation and what was it about and you will see in here your criteria evaluation and data collection types as well similarly if you go to the transcription uh you will here see the entire transcription and you are also able to see the interruptions are happening actually like when I talk over the agent itself the agent is STO speaking and stop listening to me so we are able to recreate the smooth transition of the human language and human speaking with the turn taking and interactions as you can see here and if we go to again here back you are also able to deploy this product in different formats for example if you visit our web page again you will be able to see in here like we deployed our conversation agent as a start a call as a widget similarly if you want to deploy your agent over the over your website or a different places you are also able to deploy it as a small widget and you are able to customize this widget as you want in different formats and similarly you can change the name type and the image in here uh before going into second demo I would like to check if there's any questions yeah I think we have a couple so the first one is from Bal and um it's is there any way to provide the system prompt as a templates with placeholders that can be filled out for each conversation so for example user's full name yes so we uh we have a functionality called uh conversation configurations which means basically for each conversation let's say you define an agent and for each conversation you are able to overwrite the entire prompt so let's say you want to change the username over there and you know the username of the telephone number or some information you can pass that per conversation without needing to recreate each time agent so you can customize uh your agent per conversation perfect and then the second one is uh from Matthew what is the architecture maybe so sorry I just add that like variable injection is also coming so like U one way is to like override the prom you can just do whatever you want and if you just want to kind of maybe it's a little bit safer that you you keep some PRS fixed and you can only let's say replace some part that is coming like a relatively soon fantastic uh that was news to me as well everyone so this is very good news uh so question from Matthew what is the architecture for the knowledge bases currently uh we have a actually Joseph you want to take this uh yes so basically currently the the like you can you can upload any file or URL and just copy paste anything you want and then it's kind of at the moment it's just kind of like fully feding into the agent like going forward we will deploy a custom uh rack solution which will make some kind of which will give you some kind of optionality on like okay what do you want to fit to the agent when do you want to use rack when do you want to split your knowledge base into let's say two agents like ones is one is is you know proficient in answering questions about personal loans and the other one is about life insurance right so those knowledge bases can be like safely split U and then rack can be you know integrated on top of them kind of depends on you so at the moment at the moment it's um it's it's more proficient in in in single engine experiences with heavy Fus on the orchestration and kind of voice part of the things and if you want to have some Advanced let's say knowledge based then you need to use this customer L functionality where you can integrate it wherever you want you give us endpoint and you you produce answer and how do you produce the answer is completely up to you but going forward you know it's it's very good to be like stay tuned like you know in weeks to weeks to come uh we will have some Advanced uh knowledge based structuring and and retrieval for sure fantastic um we have a couple more questions but maybe I can it and then we can take it again so in our next demo I want to show more of our tool calling functionality and how you can use the agent itself in different um let's call it formats and in our second functionality so this is my personal uh calendar and you can see like it's a bit busy tomorrow and we will be demonstrating a case where we Define this agent to handle calls as a personal assistant and also able to make uh some bookings in real life and how we configure this uh agent itself is first we are providing first message as usual and we have a system prompt where describe this is the hiat perceptionist and like people are calling to book a time and we Define two function to system prompt the first one is the get available SS basically llm understands uh to use this function when uh it needs some get available slots and we have a book meeting function where it use this function to book a call actually and it also understand it needs to get the full name and how long they want to meet and the email addresses and to arrange this function calling tool calling uh you can click on add a tool and how it's going to look like is we are using the c.com example in this case first you are defining your uh Cod tool name and give a description of what it's going to do and also the uh rest API call in here with the URL and you can add the different headers called API version content type and you can also pass your API token in this case there for using the making the calls to call.com together with it you can also add some query parameters we are passing here start type start time end time and event type ID and very similarly for the book me meeting you will see it's a post request call.com for booking a meeting and we are able to actually pass a specific objects here which will be the attendee object and this will require some uh attendee uh in like time zones email and name and Etc and just to see this in uh live demo I need to share just the tab so you can get the voice otherwise the voice is not transferred thanks for calling I am hikmet scheduling agent how can I help hey does hickmet have some free time tomorrow morning okay hmet is available tomorrow morning at 9:30 11:00 and 11:30 would you like to book a meeting at any of these times actually last he have availability at 2: p.m. tomorrow hickmet is not available at 2m tomorrow would you like to choose another time or should I check availability for a different day um the morning times doesn't work what are the availability times for the afternoon tomorrow hickmet is available tomorrow afternoon at 1:00 3:30 4:00 and 4:30 would you like to book a meeting at any of these times we can book a meeting at uh 4: P.M tomorrow great could you please provide your full name and email address to complete the booking for 4 o'clock tomorrow yes my full name is hikmet deir and email address is John gmail.com your meeting with hikmet is successfully booked for tomorrow at 4 o'clock you can join the meeting using this link meetgoogle.com mvi IQ jdfj https meet. goole.com mvi IQ JH dfj thank you very much if you have any other questions or need further assistance feel free to ask so that's the demo and just to show you how it's actually booking let me share the entire screen again so as you can see in here it actually checked my schedul T and it says okay hmet is free at 1 to 1:30 and 3 to 3:30 and 4 to 4:30 and and I was saying like don't book anything after 5:00 P p.m. similarly when we ask for the morning slots it GES to 9:30 11 11 and 11:30 and if you actually check the how the conversation looked like and how did it work in the background you can actually again see the entire conversation and transcript similarly and if you go to the transcription you will actually see which TOS tools are called with which parameters and you will be able to see the slots it return so when it's answering me it understands okay there's a 930 and SPS 11 and 11:30 and that's how it understand and returns to me similarly when I ask about the 2 pm specifically it ask for the 2 pm but there's no slot available at 2m and finally when I ask for 4 P.M booking it actually calls the booking uh function first of course it's asking my name and email addresses and books the call and as we can see here actually it booked the call together with the John Gmail and this is the meeting in wh link where it was saying to me together with the name surname and the email thises you can use these kind of examples not just as a personal assistant but you can actually use it as a your in in your e-commerce websites or you can use it for making outbound sales course inbound sales course you can use it to maybe redirect call to another salesperson there is quite a bit of things you can do and like the imagination is limit here but we provide the entire functionality here um and as you can see like it was very good at pronunciation my name hickmet is not a Turkish is not an English name and it's able to to understand how it's going to pronounce and like also dictate it because you are able to pass different keywords in here to make sure the speech to text part is working perfectly and for the text speech part you can also pass pronunciation dictionaries if you have a specific word and if you want them to be pronounced in a specific way together with this we all of the things we showed here over the UI is available and achievable over our sdks we are providing both python react ja and iOS STK together with we provide a direct webset Connection in here and we have a detailed guides on how to start in our documentation page and now I think we can take some questions yeah fantastic thank you for that um by the way I'm still waiting for uh for the guide on how I can build my own receptionist hiet uh I think that that's going to be very helpful um so we we have a question from Bal so can the voice be set per conversation rather than per agent and uh essentially how many agents can be created yes um so we can you can overwrite per conversation the The Voice you want to use voice ID the language which is going to answer the first message and the system prompt and as Joseph mentioned we will be also bringing the dynamic prompt so you will be just uh being able to add or remove uh couple of variables when you're making a conversation fantastic maybe I can just very quickly actually show where you can find it in the docs maybe then so where where you go to the docs you basically click here this Dynamic conversation which just like H mentioned and then then basically here are the overrides so Define the the overr rights for this conversation that you are starting so for example the customer bank account balance is da da da they are based in da location you can over the prompt da da But to answer your question you can also exactly override the voice ID which should be used for this call awesome um I think sort of related to this a question from Akai um considering that we can provide a single prompt have we observed any hallucinations so with some complex use cases it might be that the instructions can actually confuse the llm I can take this um so basically hallucinations of course happens in the all of the LMS but like what we are providing here is the entire functionality endtoend voice in voice out and in our audio generation and the speech to text generation we are quite stable depending on what kind of llm you are using you can be more stable or less stable let's call it um so far we see in all of the big models and from the big providers they are quite stable and we will have some also release some guidelines around how to make strong prompting to uh to make sure like the Hallucination is less or like not happening to put some guard rails over there fantastic um a question uh from Paulo is it possible to connect the agent to let's say an instant messaging application like WhatsApp for exchanging these messages via well voice message Joseph you want to take it or you want me to answer yeah sure uh so yeah this is this is uh actually this this is what people also ask or our our our docs agent uh you would know this exact question what's up question so here we have we have a com we deployed our on com on our docs page so you can you know ask for help on docs but it's a little bit slow because it has very lot of data but basically the answer what already people were asking uh this agent is to use the business WhatsApp um uh yeah WhatsApp business API so basically you need to use the python SDK so our python SDK which will receive the from the business WhatsApp business API you will receive the audio from the call you will forward it to us via this python SDK and then when we send you audio back that should be played to the user you then forward that part to back to the WhatsApp business API yeah just to show you how it's going to work out also basically uh if you check our like python STK in here we detailed how to use it specifically and like you will see the conversation start and Etc and even if you want to go more bare Bond we also provide direct website connection and you can redirect audio in audio part from this webset and these are the five different events we will be returning with the web second yeah fantastic um Josh asks can the AI speech rate be adjusted um potentially some of the older users might struggle um you know keeping up with the faster speech you want to take this B down or you want me to answer I mean I can I take it so at the moment this is voice dependent uh we don't have uh quite yet a way to um change the rate of speaking um so I would say it is something that is that is coming uh likely next year but stay tuned yeah one couple of like suggestion here like we have like I think over 3,000 voices at this point in the voice Library we have like Fast voice speakers slow speakers narrative and Etc I think there is a voice for every use case pretty much on top of that you can also bring your own voice with the professional voice clones PVCs and we are also supporting conver in PVCs in the conversation and uh I can see we have a couple of questions around the voices themselves so basically can we uh you know Andrew asks can we use custom voices for the agents uh another user asked if it's possible to use your own voice as an instant voice clone um to to power an agent yeah U we generally suggest using either the default voices or the professional voice clones because they have a higher quality and the processing times in the in the inference is a bit faster uh however uh instance voice cles we allow ICS to use as well but like we are suggesting the pbcs perfect um another one from akshai um I'm more interested in latency as it is the biggest blocker so what is the average latency in this case including tphy time can you guys uh give kind of like a breakdown of St llm TTS and telepan time Joseph you want to take it or you want me to take it sure you can maybe then add a little bit on the telephony part um so generally speaking generally speaking like once we recognize that the you know the user is expecting an answer and we start to generate the answer uh including of the transcription including of the all the turn logic including the llm streaming including the TTS streaming it's around it's around 900 milliseconds when you add the the silence that you need to recognize that the user kind of is expecting an answer you get to around 1.2 1.3 seconds uh we we we are kind of like some improvements on the on a significant improvements on the TTS part that coming which will cut this down by one or 200 milliseconds hopefully and uh so so that's kind of generic latency where you stand then when you want to use the telephony you know you need to add the the kind of the audio for for forwarding part but now we will have the native solution so basically then it's just like kind of like how much latency is added on if you use twio and the tro is only based let's say in US servers and then the caller calls from some in some some Asia or something and it needs to go first to trillio and then from trilo to us and then from us to trillio and then from trillio to them it may add a little bit but if if if let's say the color is in the US I think it should be it should be actually pretty pretty good yeah we tested yeah we tested this couple of in ourselves and also with the clients overall the t t added latency will be maybe like 100 millisecond or 150 millisecond let's call it this is mostly caused by the T and the network latency but overall experience is still smooth and realistic I would call and everything includ that we are still around like 1 second Benchmark or like maybe 1.4 second Benchmark which is like and this is just to say as just mentioned like the moment you stop speaking the moment agent start speaks again so this is like literally silence to silence to speak thank you for that uh related to to you mentioned the the kind of the aspect of realism Josh Josh is asking if there is a way to incorporate background noises such as a call center Ambiance or just make it feel a little bit more realistic yeah um this is something we are also working on adding natively but currently you are able to do this very easily so we are providing the webs connection both Wes basically and this webset connection you can easily overwrite or add another audio part in this web second connection just to play a u a background noise in Loop basically like we will put a guide for this out soon as well this is some of our clients are doing fantastic and then um two questions slightly related so one is is it possible to do bulk calls and the flip side of that is is there way to prevent spam basically if a competitor is booking let's say hundred uh hundreds of meetings with your emplo employees just say you want to take this yes so bull calls do you mean like out I guess that's outbound calls right uh so uh yeah so so for out calls I mean the the only problem there is that like depending on where you are located like there are different kind of legal legal legal obstructions that you need to say like if if let's say it's a user who filled in some form and then they tick that the AI agent will call them back it's probably okay if they didn't explicitly say that it's it's likely not okay at Le in the US there's problem there but then you should also kind of like in the first message if you call someone like say that this is an AI calling or stuff like that but in general in general yeah in general it's not like a it's it's not it's not kind of a big issue like the only difference between an incoming call and an outcoming call is that when you do an outcoming call the user starts to speak first because they pick up they say hello so there you should just set the first message empty then it will the agent will expect to to to for the user to speak and you can say for example in prompted after you received the initial message responded hey I am XX calling from y why and are interested in that that and we may probably provide some na functionality in the future we upload some some numbers and we can like do those calls for you uh but yeah in in in in the meantime yeah in the mean time actually hi do know do you know what's the best strategy to initiate those CL can this basically uh in here we also have our AI safety and regulation teams internally and we are actively monitoring and checking so like of course like general terms of conditions for the Lums applied to conversation AI as well so any misuse or malpractice we will be having actions on top of that um together with this uh yeah we provide the functionality in this case but like regulations and the laws are changing from country to Country it's best to um work with your own legal council in internal I guess perfect and on the and also yeah and just to add on the on the the limits on the limits per agent yeah this is this is something so we have limits we have some limits like per IP and so on we have somebody would want to do some like Advanced uh you know misuse yeah we can we can we can we can definitely we should definitely support some kind of maximum number of conc calls per per agent perfect uh I believe uh aai is talking about 50,000 calls per day in his specific scenario uh so maybe it's h it's something that we can also take offline yeah we can take it offline but as jph mentioned we have the concurrency and rate limit limits and these limits are like increasing decreasing depending on the um Enterprise agreements basically yeah perfect and uh when it comes to spam prevention yeah let me let me just remind you guys is there any way to prevent spam so for example a competitor booking hundreds of meetings with your employees so like what we are providing right now is the entire conversation AI functionality let's call it and this functionality can be used for different purposes like if we detect there's a Spam usage we will of course take an action on this but we are not actively uh let's call it uh monitoring or checking like which what each uh implementation is currently doing so to say and like yeah for each call users needs to like work like basically like run with us and like basically spend money on with us and Etc it's not very easy to do this and like we suggest like if you are doing this on your side you can also put rate limits on your uh implementation Roger that Roger that um okay so uh let's see we have a couple of additional questions here so to is asking is it possible to deploy my own fine fine tuned model I suspect he's referring to an llm model yes definitely and currently we are supporting I can also show this very quickly so how you are going to do is basically if you go to the our web page sorry in your documentation you will see integrate your own model this is specifically first of all if you want to just use the custom model but bring your own key this is very straightforward if you want to deploy your own custom llm server we have a simple server case in here just to run it and then what you will need to do is configure that server endpoint basically where we need to make API end calls together with your API key um you can still arrange the temperature and the token usage and Etc and on top of this you are also able to pass additional parameters to your llm so let's say you want to pass some specific user IDs or some hash IDs or something like that you are able to pass this information to LM and when the conversation is happening we will Echo back these uh responses so say and you will be able to see in your response llm response 11 laps extra vales perfect okay I don't think we have any more questions um let me just check the chat maybe a couple of people have uh asked it there um yeah Ferrera uh Christian is asking how can I integrate my tools like functions that I created in a repo um sure we also have a as I showed in the calendar example it's very similar how we are doing in the calendar example we ask you to provide a basically an API handpoint where your tool is located and uh basically we make a call and we Define that end point one is going to need to be HD it and also we have a detailed example on our documentation again about how to set this up we both support external tool calling and the client functions as well fantastic okay um I don't believe we have any more questions so thank you both ever so much for this uh present uh maybe just just one more thing before before we close it up that also you can you can on in our docs you can find our Discord channel so you can also feel free to join the Discord Channel there is a conversational AI threat we check it regularly if you have any technical problem or issue you can report there and we are very active and you know try to resolve the issues and you can also you know connect with bdan and with the team uh to to have any kind of like more custom Enterprise uh you know Solutions or or or personalized approach in general you know we are super happy happy when you when you break our to break our products you know like please please do so we hope it's it's it's it's actually not possible but if if so you know we are generally very active and we want you to to be successful with the with the products that we deploy and we we are super happy to receive any feedback on on any future requests and you know if they're reasonable they they will get deployed stole the words out of my mouth Yosef um so uh yes once more please play around with it uh read the dou umentation ask questions reach out to us um and thank you ever so much for your time thanks everyone thank you bye bye bye thank you byebye