Only Gemini AI can DO THIS!

Key Takeaways

the most underrated Gemini feature is something that no other llm or chat bot can do at this particular Point 11 Labs recently launched scribe and everybody was going bonkers because it can do transcription with speaker diarization which means if two people are speaking in an audio it can just separate a speaker one and speaker 2 and this feature is just hidden inside Gemini and some of Gemini marketing team doesn't even talk about it so in this video I'm going to show you how you can use Google AI studio with Gemini and then you can do audio transcription including speaker diarization and then do much more than that to start with I've got my Google AI studio and I've already uploaded the video of Satya nadala appearing on a podcast and this is a 4 minute 57 seconds clip and then as you can see here the first thing that Google AI Studio did is it took this video clip here converted into tokens for llms everything is token okay so it took this video and then it understands the video it understands the audio behind it and then it converted into tokens so this has taken 8 87,000 tokens 616 tokens so this is basically 87,000 616 and the model is 1 million tokens which means we can pump in a lot more video content inside it and all I did is I just went ahead and then said can you give me the transcript of this and I guess you can do it separately for separate speakers thanks and I didn't even mention how many speakers are there and this is the solution until this point I've tested it with only two speakers but it'll be pretty fascinating to see if it can do three speakers as well and also if you are like an EML nerd machine learning nerd you can do this kind of stuff even using cin clustering like if you have studied cin clustering if you have use cman clustering for customer segmentation or different kind of task you can do speaker diarization using cman clustering I think I might have a video where we did it for whisper but for now the llm can do this completely for us upload the video just say this thing and then the llm has done the transcript for me and then it says of course here is a transcript for the video organized by speaker and then you can see Satya speaking and then there is host then SAA then there is host and then do everything one important thing with these llms is that you have to always make sure that they're not hallucinating because I've had this experience where I've given this to like different LMS and then sometimes the time stamp is completely rubbish just gives you a time stamp but it is not sure so what I did is I wanted to check this out so I've got the podcast here it's 4 minute 57 seconds and I'm going to go here and then just take one clip where Satya appears so at 2:30 Satya is there like m so I think s companies can okay so I'm going to go go to 230 here so 230 here just before 230 we'll stop so therefore that's the contract um so I think yes companies can you know so as you can see here um I think yes companies you know have so that that Gap where he paused and then Satya again started speaking is even classified or categorized as like two different instances with um with the SR with the subtitle here or the time stamp here and this is extremely helpful if you're trying to do like the Tik Tok Style dancing captions or you know the zooming in captions this is extremely valuable for those kind of cases as well so I'm going to go back here and then find another time stamp where instead of Satya the host is talking so I'm going to go and then say Okay 439 the host is asking if you did leave Microsoft what company would you start that's a that's a very interesting question 439 is it is it 439 let's see yourself if you did leave Microsoft what company would you start okay that that is there and then the answer is company I would start man uh like uh I don't know what does he say company I would start man like that's where the company man and my it's amazing company I would start man that's like where the company on okay great so this you you saw how it was done right um but I want to do it live so that you can see how much time it takes and how the exact process is open your Google eii studio if you have never Google used Google eii studio I've got a separate video where I just cover most of the things this is not a use case that I covered in that video and some of you had given me feedback that I should go deeper more um Advanced so here is one Advanced use case so click create a new prompt and after you click create a new prompt you can select any model that you want but I would go ahead with flash 2.0 cuz flash 2.0 is one of the fastest model available but also it's really really good and if you want to translate this into code as well like a actually working code then flash 2.0 would be cost effective for you so keep flash 2.0 so you've got 1 million token totally and all you have to do is go here just drag the podcast whatever the clip here is and then paste it here and then that's it so now all you have to do is you have to just go say um and as you can see here after you upload the video clip and in this case it's not even audio you can do this with audio clip alone you can do it with audio plus video clip you can just do it with video clip but without audio but take it as frames so you can do a lot of different things so here in this case the 3 minute 18sec clip it has 58,000 tokens where wherein like the previous one had uh 7 87,000 tokens for a 4 minute 57 second clip so you can understand how many tokens you can just like pump in inside the existing context window you can do the same thing with 2 million token model as well I'm just giving you a demo of how it can do faster with Gemini 2.0 flash okay so now all I'm going to do is can you create a spe uh audio an audio transcript and um um you can also do it I'm not even saying that speaker diation right you can also do it by Speaker by Speaker can you give me a okay me a Json output let's see if it can give us a Json output that would be pretty fascinating something that I've not experimented before just running it for the first time so you could expect that something could go wrong so now you can understand how much time it took to process a video clip but also you can see how much time it is taking okay because I asked for Json then it is not giving me the time stamp okay so I'm going to go ahead and then say um oh sorry um the time stamp is missing bro I hope Google doesn't mind me calling it bro um but yeah if it Minds you can see that it has taken 8 seconds um it did apologize and we have um multiple time stamps so we have got 0 0 where the host is asking a question 010 where Mark is answering and 125 is where the host is answering asking something again so I'm going to go back again then see first of all 0 0 Let's see like big picture these next couple of decades sure what's happening with AI um and where is Mark answering Mark is answering at 10 okay so we're going to go to 10 9 the course has his I think it's going to be pretty fundamental I think it's he said I think it's going to be pretty fundamental very nice and we going to see 124 somewhere like 123 and then see if he ends with this statement which is I think I think it's going to be I I I does he really say I three times let's go and see so I'm going to go here 124 I think it's going to be it's it's going to really enable people to do the things that they want a lot more is just my view but it will really change the way that we work and give people all these creative tools to do different things that they yeah I I think I think it's going to be okay so did look like he actually said I I I I like that so maybe um we can trust J in this case and 12 125 one second the host is speaking they want a lot more is my view is it your view that like on a great so the experiment the live demo actually worked and like I said you can do a lot more other things for example now we forced it to give us a Json that means you can pass through this you can probably take only host one you can take only the other speaker and in fact like you will be pretty fascinating to know that it figured out that this is Mark Zuckerberg I didn't say it probably the vision language model understand this is Mark Zuckerberg and the same thing like even um you can see okay maybe I said Satya I don't know if that is a clue here but it managed to figure out that this is saaad and here in this case this is Mark zabur which I've not explicitly given unless until it figured it out from the file name so you can do this stuff you can probably take this and then summarize it um for example I can just go here and then say um can you take one what SN said and summarize it as a tweet so the potential is immense because you're not dealing with to only an audio uh let's say audio to text model speech to text model rather you're dealing with a multimodel large language model which can do multiple other things so for example here in this particular case we can take what seya said and then summarize it as a simple tweet and all these things without you having to pay any Penny I mean like I know now somebody would say that the data that I'm using is what Google is going to use but if I'm going to get jav 2.0 Flash and a bunch of other models for free where I don't have to pay for 11 La subscribe or descript or bunch of other tools which podcasters exclusively use just to do this particular bit that I'm telling you then you know you don't need an editor you don't need a podcasting tool all you need is Google eii studio and a very good model like Gemini 2.0 flash I don't know why Google is not talking about it a lot but here you go Google thank you so much or I should say Google you're welcome see you in another video Happy prompting