Try Google's Gemini Pro and Pro Vision via API in Python

Key Takeaways

hello everyone welcome to AI anytime channel in this video we are going to look at Gemini API okay so Google has come up with their Gemini family models like Gemini Pro Gemini Pro Vision and Gemini Ultra is also on the card right they will release the that for that very soon as well now the apis are now available to leverage these models in your applications uh that you will be building and these are available through uh different mediums one being through Google AI studio and the other one being vertic AI on gcp Google Cloud platform so studio is more of like when you use open AI the playgrounds and using the API from open Ai and just imagine that how open AI has deployed the models within auu uh that's called aure open AI where you basically for Enterprises where there are some concerns about the data privacy Protection security copyright infrigments and whatnot right so if you really want to build a scalable solution at the same time leveraging different Google products and services the better will be to use vtic AI uh models and of course to use it through vertic AI because then you'll be using the know vertic AI notebooks the different Services of Google like Cloud run the functions uh the containers etc etc right so that's a that's a way of doing it the other way if you really want to test it out and build some proof of Concepts or faster prototyping better to use uh Google AI studio so this is going to be a playlist where I'll be covering only the Gemini models uh from uh like introduction to building rag pipeline so uh there will be around four or five different videos in this playlist in this video going to start with the intro that how we can know leverage these models through the API without any orchestration Frameworks like Lang chain and the Lama index index of the world we're going to use you know couple of your prompts one is like putting Simple Text prompt and the other is to uh trite out the vision capability of provision model of Gemini and we'll look at a software development use case where we're looking at to generate or create test cases of the application screenshots that's what we're going to do in this video on a very simple example and in the upcoming videos we'll look at uh rag pipelines uh using orchestration Frameworks and Vector databases and stores now you can use the apis for free which is fantastic I mean you know Google is trying to create some competitions right now with the uh from their com from a commercial standpoint because they have given some uh some free options where you can leverage this apis in free but of course if you want to uh use it uh for some n number of users then then you have to basically look at the other options where you have to pay right so let's start now uh experimenting with Gemini apis you know I'm going to use a notebook and going to just take the API key and see how we can just put some prompts and generate some responses so let's start our experimentation with uh Google Gemini Pro and provision model all right so to experiment with Gemini uh in this video that we're going to do is look at the intro of how to use the API and just to make some API calls with some prompts to test Gemini model the second part of this video will be more of a rag implementation where we will use an orchestration framework you know to basically orchestrate the rag pipeline okay so we going to use llama index here in this video this is going to be a long video but trust me this is going to be a very very informative video for you this is the only video that you need for Gemini at least for now okay the a lot of development that will happen and of course I will be creating a lot of other videos as well now let me just rename it as Gemini you know demo or something like that okay now uh but to use Gemini you know you need an API key you will get the API key from here you have to come to this Google AI studio and you have to see in the left hand side that have something called G API key if you don't see anything here you have to create API key new project and then just copy it once you see that I will give the link of this studio in the video description now that's all here what you have to do is you have to install a few thing so for now you have to install just this letter we see what we have to install with the llamas and all now just come here and you need this Library that's called Google generative AI okay this is the library that you need already already have installed so I'm not going to install this so the first thing that we are going to do is let me call this as a markdown and here I'm going to call this as with directly via API without orchestration framework okay without orchestration framework perfect and let's do that so what I'm going to do is first I'm going to write a uh wrapper function or a or a utility function for markdown output that's also is available that's what you know they also suggest now so let's do that so import path La uh import text WP uh import text WP and then the Google generative AI Library so Google do generative AI as jna and then let's have couple couple of display if required you know for the Imaging and all so IPython display asge display excuse me this is small only display uh the next is and this is ridiculous by the way okay uh so this is not consistent in the uh definition of the functions or classes you see the dis display start with small and that starts with caps you know this is not a right way of creating these libraries but anyway uh sorry this is not IPython as it's going to be import okay uh once you do that you see that we have already imported the library the next thing is to define the markdown function so let me just do Define toore markdown and then here I'll will pass my text so basically to beautify the answer that we generate uh from Gemini model okay so text. replace and here in the text. replace let's replace dot with you know uh like a stct or something like that for now okay uh and then return we're going to use this markdown and here I'm going to wrap the text so text WP do indent it's a method and here I'm going to pass my text not text WRA okay and then let's use a predicate so uh predicate and once you use the predicate equals I'm going to use Lambda and excuse me there's something wrong here okay Lambda and then this becomes true let's do that okay now we are done with this guys now what I'm going to do next is I'm going to basically just Define the API keys I already have here okay so let me just use that just to save some time already have written this is my API key of course I will delete the key once I complete the video now let me just import it once I import this okay what I'm going to do is I'm going to first show the model so let's load the model so gen AI do generative that's the that's the class generative model okay so this generative model and here you have to pass your model you can find out the signature over here the init signature that explains it takes a model name as an input which is the string value and here I'm going to pass my Gemini Pro so both Gemini Pro and Gemini Pro Vision is available for that you have to come here in create new or let's let's go to uh free form prompt and here you can this is basically a playground here you can select your model so you can see pro and pro Vision are available so I'm just going to define a pro here for now and let's run this now once I run it uh you'll see let me just do a model now once you do a model it will print the very high level pipeline generative model model model name of this class and then generation config you can Define some config that we'll look at in a bit okay and now let me ask a question so very simple response equals model. generate so model do generate content and I will explain there are different uh way of generation is available like async Cronus streaming and so on and so forth right now let me ask what is AI now once I ask that the next question the next thing that I have to do is if you do response now let me do response once you do response by the way you it will just show you the object now what I'm going to do is response. text we have to do response. text you will see some output which might not look that you know I mean that beautified an output right is it's difficult to read and it's also difficult to interpret right so what I'm going to do is let's use that function to markdown response. text and it gives his name text WRA is not defined I don't know what the mistake that I did somewhere okay text ra this is wrong okay let's see if this works otherwise we'll just keep on going can see now fantastic right a beautiful answer guys you know you see the markdown what and because we are working in collab right if you're working this in some ID there's a different way of handling this because we'll be then working on some flask of fast API back end we'll also do that in upcoming you see it says artificial intelligence is the theory and development of computer systems that perform tasks normally requiring human intelligence such as visual perceptions blah blah blah blah right so you get the output here now this is the one way of doing things now let me just also execute this is one of the question you can keep on asking different questions now I will just uh bring the vision capability so let me just write here Vision capability and in Vision capability let's do import P ah I don't know what just happened this bad auto correction here guys okay import p. image and what I going to do now let me show you now I'm writing IMG equals P do image do open so I'm going to open an image here so let me just uh show you what I'm going to do let me open my collab and here I'm going to open okay let's take this so what I'm going to do is I want to take a screenshot of the visible part of this image okay let's create test cases now if you wants to look at the developer productivity in your organization and if you want to build an automated system that helps you generate test cases and multiple edge cases within that test cases using gp4 or for Gemini now because it has some Vision capabilities let's try it out you know those kind of huge cases why not right so let me just rename this as something like you know uh you know input it's fine okay let's save on desktop where I have this I'm going to call in Gemini Rag and that's it now let's see if it Ables to do something this is a real testing guys I'm not a big fan of you know the evaluation benchmarks okay so uh now let's come here and let's see if we have you can see now we have input. PNG so let's bring that input PNG here you can see it's the source and let me just bring image here it will you know show the image in the notebook you can see a beautiful image over here of Google AI studio now we have to again bring the vision model so let me just call it Vision model it has Vision capabilities you know gen a. generative AI model again generative model and inside this model I'm going to use Gemini provision okay so let's call it Gemini provision okay and you can see it gives you Gemini Pro Vision By the way response equals then use let's use Vision model so let me just use Vision model dot generate content and in generate content I'm going to pass my IMG okay so let's pass right now what what let's pass this uh what what was the variable here IMG okay let's excuse me let's pass this image for this and let's see what we get here okay we'll see that and let me just print the response for now again response. text and let me just do that it says the assistant of the left is showing so basically it explains that image it's not explaining it that's what it does here by the way now explaining is sometime you know something like if you want to ask a question right uh in that case how will you do it okay uh and that's something that we have to look at so let's do that so it says the assistant on the left is showing a list let me bring the markdown thing again so to markdown and let's pass response txt or text by the way and you see the assistant on the left is showing a list of options the assistant on the right is showing a black screen with a loading icon close not that good not impressive okay uh to be honest not impressive okay uh we will see the custom prompts and all now what we can also do here uh let's try it out Pro probably let me just do a response equals here I'm going to write model. generate content and in generate content what I'm going to do is here I'm going to probably Define something like you know generate uh test cases after identifying all the components uh you or let let me write like this you are uh you are an automated software tester at an organization at at at an IT company okay now generate test cases after identifying all the components okay now let's keep Simple here and then cool okay this is fine uh and after that we have to make a comma here so let me just do that okay now after making the comma I'm going to pass my IMG and then uh after here let's pass the stream because we have to make the stream equals true Stream True and then response Dot and here you have to use resolve not the so here you have to use resolve you can see it says resolve that's the uh you can see it signature dock string it explains there's no dock string in that and then we let's write to markdown response dot uh text fine let's try it out it says let me see what it says 400 image input modity not enabled for models G Pro okay image input modality is not enabled for models Gemini Pro ah okay okay so we are doing something wrong uh probably the model then it's overriding uh OKAY model it should not be model it should be Vision model my bad that's why I write it because it's now taking Gemini Pro the earlier that we defined okay that was the mistake uh now we have Vision model. generate content and in this we are expecting that it will generate some output for you and let's figure it out if that out because I have worked on a similar use case and you can see input hello world text input fantastic you know a component to model component through safety settings you know advanced settings okay it gives you some response you know for the test cases of course we can you can put more custom instructions to look at the output but you know very impressive I liked it you know uh the uh nature of this that it has suggested uh very impressive now let's find it out and you can see that Vision capabilities we took an hdlc use case software development life cycle where we are generating test cases out of your image this is your image now the next is uh let's let look at how to find all the models let me see if there's a way of doing it okay so I think there is something called generate supported generation method uh so let's do a for Loop so for M which is nothing but the model in J ai. list models yeah you can see that we have something called list models what if I just do j. list models will it do anything okay it's an object uh what if I'm doing print the same thing hopefully okay fine so uh we need to use this in this way so let me just do so for m in gen ai. list models and then of course it's a for Loop so now in this Loop I'm going to say if generate content uh maybe I have to see uh this part I'm not sure about it but let's try it out in M do supported generation supported yes I was expecting this supported generation method let's print the M do name you can see right now we have two models available models Gemini Pro and models Gemini Pro Vision okay so you can find out all the available models like this okay so I think we should you know uh stop this video here guys for the first part of the uh this video the playlist that I'm creating right now so in the in the op coming videos we'll also see how we can you know use the orchestration Frameworks like Lama index and Lang chain you know to use Gemini for the rag pipeline okay so I already have recorded that those videos but I think creating a leny video will not make sense so maybe we can have three or four videos in a playlist and that will be a that will be a better way but I hope you can now look at this this was an hdlc ug case where we look at creating test cases for the given image or app application screenshot and this is what it you know you can feed more custom instructions with some you know few s prompting as well to make it more better and know other response but this is what we did you know we tried it out without orchestration framework directly via API that how we can take an API and test it out on some questions some Vision capabilities and also how what all models are available into it so wait for the upcoming video uh guys this notebook will be available on my GitHub repository if you have any comments and thoughts related to this please let me know that's all uh for this experimentation guys I hope you uh understood that how you can basically uh you know leverage these model through an API and that to in free at this moment uh and you know the The Notebook will be available on my giory and the more videos will come very soon I will start uh posting them uh in the next couple of days with the rack pipelines that where you can use Gemini and I'm also going to create a video of GPT 4V versus Gemini Pro Vision and we'll see what are the differences and which model is better at least on few hug cases now if you have any uh thoughts feedbacks comments please let me know in the comment box you can also reach out to me through my social media channels please find those details in the channel banner and also the channel about us uh if you like the content uh please hit the like icon and if you haven't subscribed the channel yet please do subscribe the channel share the video and Channel with your friends and to peer thank you so much for watching see you in the next one