The Gemini API: From prototype to production

Key Takeaways

welcome to the session I hope you guys are having a good time okay perfect uh so in this session uh we'll talk about a lot of interesting stuff we'll do Hands-On um I'll do the handson you'll see um but the essential uh idea here is to see how we can use the API to do something which is sort of more idea to prototype like sort of um sorry prototype to production and that's what we're going to do uh myself L Nigam I'm a developer Advocate at Google Cloud um and let's see what we're going to talk today so the first thing is uh we'll try to create a Persona of a developer uh me being a developer I would like to sort of do it in this way where um I take a problem I think of myself as doing exactly the same thing which every developer would be doing and that's exactly what we'll do we'll try to create a Persona we'll try to in introduce some of the Gemini models um and then we'll see some of the requirements from a leadership perspective from a developer perspective and then we'll go through bunch of notebooks three essentially and then the last part which is my favorite where we'll try to sort of try to extend the idea into more sort of production ready uh things more on that later so let's start with the developer Challenge and like I said I'm going to take a Persona so imagine that I'm working in a financial firm and the boss comes to me and says hey we want to automate bunch of stuff that let's say analysts are doing and how can we do that what are the things that we can help them achieve in that sense now the problem statement that has been given is let's say we want to sort of reduce the research time and cost for the financial analyst team and to do that um what we need is we need to solve this problem from a multimodal perspective because if you think about the work that analysts or researchers do they essentially go through so many different kinds of sources they go through reports they go through uh PDFs um videos audios earning calls there's so much modality sort of involved in this and essentially that's what our problem statement is at an idea stage I would want to sort of prototype this and see whether I'm able to build something quickly right and of course as with most sort of prototypes you're not given too much time and the flow that I come up with as a developer is that essentially I need to start with a source document and this could be reports Company videos earning calls images bunch of those things right and then what I should be able to do is I should be able to ask questions uh irrespective of the modality so things like what was the revenue what's the analyst question asked to specific person how is X product doing and so and so forth and all of that uh should be able to give me back the answers now of course as a developer the first thing is we all do research we go back to the Google we try to figure out what are the different models that we have and in this case uh for obvious reasons I'm going to come back to the same problem statement which is I need to sort of validate the multimodality um of this problem statement so I need to sort of understand how can I take all of these modalities get to the answer quickly the answer of course in this case is Gemini Gemini has sort of these different layers where you can pass these data points it's buil sort of responsib um it has sort of different choices in terms of models which you all see yesterday um so that's pretty much what we are trying to do but the most important feature here is multimodality like I've been saying so many times and you can count the number of times I'll say this in the overall presentation uh but essentially that's what we need the the core feature beyond anything is the contextual Windows of 1 million and the multimodality that we need to solve this problem from a validation of the Prototype stage so before we jump on to The Notebook where I show you how we can do that uh let's quickly sort of summarize and this is again a very something that we all face as a developer we always have requirements from the leadership that hey we don't have budget it should be quick cheap uh there should be a lot of process across the flow that you need to have which we just saw in the previous slide it should be simple enough the scope should be reduced and from a developer perspective again the cost should be less I should not spend too much time building something it should be very easy for me to sort of build those things I should be very easy uh it should be very easy for me to validate the ideas that I have and then of course the documentation the cookbooks and examples bunch of these things right so that's what we should keep in mind when we look into all the notebooks that we'll go through today okay let's dive right into the notebooks let's get to the demo part all right so like I said said I have three uh notebooks and I'm going to sort of Zoom this out so that you can get and maybe take a uh picture if you want to see the notebooks uh I'm going to show this three times overall because we have three notebooks but just in case if you want to sort of see all of these notebooks it's hosted on my GitHub um I've also given the quick links for the cookbooks go ahead sort of explore those things as well okay so again do remember we are trying to sort of validate whether I can help my boss uh create this prototype so I'm going to use Google AI for developers and the Gemini API that we have let's see the uh sort of very basic step step by step so I'm going to install bunch of librar especially the Google Generator we SDK which is important this is what we're going to use once I do the installation and again um I have ran all of these steps in the morning so these are all pre-cache but the notebooks are available online you can go back run this and you should be able to sort of run everything as it is the next thing is I'm sort of downloading a set of data now this data is very interesting so if I go and show you this is the initial set that has been given to me so if you see these files these files are essentially bunch of PDFs MP3 videos and audios and things like that right so this has been given to me as my initial test that hey if you can ask questions to these files then we're good like we we're good to sort of prove that you know we can do these things very easily so I essentially download these files and all of this gets downloaded into my collab uh which is to say the runtime that I'm connected in the back then I essentially import the libraries uh I do the authentication which is again a very standard things you can get your keys in the Google a studio configure it load it so that you can access these models then I do three very simple things um in terms of sort of loading the models I first import all the specific uh classes that I need I Define my config and this is how you can Define your config I'm taking making the temperature one in this case I can Define the safety settings because again A lot of times especially let's say you're in a psychology kind of a domain or a healthcare you might get into some troubles where things will be blocked by default but you do have control to sort of change those settings and that's what I'm doing I'm saying and there are sort of four different levels you can refer to the documentation and then something that I'm doing is I'm importing the model so I'm now importing Gemini 1.5 Pro latest very important for us to keep this in mind the namings of the model because when we'll see we'll jump to the cloud side of things the model names might change and I was trying this was not originally part of the demo but I just got uh really excited and I thought why not add flash as well so I'm going to show you a couple of examples of flash as well just to sort of compare and how things sort of work but as you can see it's pretty same like there's nothing change except the name so we are importing these two models the next thing is I've written some helper functions which essentially does very simple things which is you take a PDF this is the blog the blogs typically and let me sort of quickly show you this so if I open that Blog the good thing is this Blog has combinations of images so something like this graphs and architectures um and we'll try to see how we can leverage this because all of you know that llms are great at doing text extractions and doing things but how about sort of reasoning with the images which are present right so we'll try to go there as well we'll try to see how we can do that but as the first step I just want to make sure that I can do the text part perfectly so I have written a simple function which goes through a PDF extracts all the text saves it in a uh simple return statement and then I'm just sending it to the model and I'm asking a simple question so as you can see I'm asking what are the key achievements for Google cloud from the following blog mention and bullet and I'm using a rich markdown here so this is not the um the actual output this is the markdown uh display of the output and you can see that it was able to sort of give me and the funny thing is it actually just took 14 seconds and this is again this is not the flash model this is the main pro model it's pretty good like I you can actually go and see whether these details are correct or not I have taken that pain and done that and all the details are pretty much accurate it's sort of quoting all of these things uh from the document I can do the same thing because the code has been written in such a way and it's so simple you can just see there's nothing going behind you can see that I'm just changing the model flash now so initially I was calling simple model which was 1.5 now I have model Flash and you can see that it still gives me the output but this time the time is sort of almost the half of the probe model so that's what we were trying to sort of uh uh do which is we want the models to be faster so if you have use cases where things are not sort of very complex you're not trying to do very complex reasoning you can just sort of send it to this and you'll get the answers um uh pretty fast and again this has 1 million tokens so there's no difference in terms of tokens okay let's move forward I have another file which is a uh ply report for Google and I have bunch of questions it's it's not restricted to asking one question I wanted to see whether I can ask multiple questions or not so I use the same function as before and I say hey can you take this PDF extract all the text and then sort of give me something by the way this PDF is almost like 4050 pages so you can see this is where the long contextual window sort of comes in handy and I'm not I'm sort of trying to be very greedy because I'm saying hey I have all of these questions can you sort of answer all of them in one go and most of these questions if you notice they are essentially very financial analyst questions like that's the kind of question they would want to ask right and again I do the same thing and the moment I hit enter again it takes sort of 14 seconds goes through bunch of text and then gives me the results in a very proper format not only that that um one of the very interesting thing which I observe quite frequently with 1.5 is that sometimes it gives you that hey uh you were asking something but it's not sort of given directly so it tells you that hey for me to answer that um I would need extra detail or you would need to provide some extra sort of context and that's what you see towards the end it says if you want me to give you the comprehensive understanding can you please provide me all of these things uh as well you can also quickly check all the different models because it's not just the two models that you have but there are many different models that you have there so you can just sort of do a simple geni do list models and there you go you can see that you have different versions of Gemini you have text embeddings and so and so forth okay let's move forward so there are two things that we have validated let's see if we can do the something uh more complex like an audio so here I have earning calls um again these are sort of videos which you see on the YouTube but they are sort of recorded calls calls and we would want to ask questions to the audio directly I sort of load this and you have a very simple function which uploads your mp3 to the cloud um so that you can sort of directly send it to the model uh which you can delete by the way so right after you get the answer you can sort of delete that and then again I ask a very specific question and these are the question which I love because I'm trying to be very sort of not direct so I'm saying what are the major Commons by Sund and Ruth I'm not mentioning their full names because I'm assuming that it should have intelligence enough to sort of pick the these things um and I'm also sort of trying to be very confusing I'm asking so many different questions and I'm trying to sort of compound everything together so I'm saying what were the names of those analysts give answer in bullets and what were the key questions that they ask if you see the answer it again this is processing the whole 1 hour audio it took 49.1 second roughly and then it came up with all of these things which I was asking so this is what Su P said AI this is what Ruth pad said on AI and again these are the names of the analyst the question that they ask this is fabulous like because imagine again every time you see this imagine that this is what that analyst would have to do by going through that video for one hour and then making note and then sort of summarizing the whole detail but now they can just do that so for me this is good because like I said I validated another modality which is the audio part let's try flash because we said it takes less time um so the next step I do is I actually try Flash in this also you can see that it took slightly lesser time it's 29.6 seconds um and then it again sort of gave me pretty much most of the same information and that's where you start seeing slight differences like if you're looking for very comprehensive detailed reasoning oriented you would prefer and you don't have latency sort of issues you would prefer 1.5 Pro if you're like no no I want the quick answers I don't care enough this will be very important when actually go to the production side all right so far so good so um like I said I was showing you that PDF where there was a lot of images and I want to also sort of check whether that is a possibility or not so what I do next is and there are different ways of doing it uh this is just one way that I thought would be interesting which is I take each page of the PDF use it as a image and then I set all these 20 Pages as images to Gemini and saying now can you answer my questions so it's very different tricky thing you're not sort of asking the text you're essentially asking the image and each page is an image because U I'm saying hey uh if there are so many graphs and everything why not sort of do that and again I've have written a simple function go ahead see those functions they're very simple they're not complicated um and then I ask again a very specific question I say what is the EMF U and bf16 blah blah blah so many different weird things that I'm asking the interesting part is I'm asking it to side the table and page number and explain the significance of the results so this is where I'm sort of doing two things I'm asking it to site so that I know that it's not hallucinating and the second thing is I'm asking it to sort of reason through the whole thing and give me why this is sort of significant again it took sort of 42 seconds uh but um it tells me exactly the things that I was looking for that answer is not directly available in the given document and it sort of clearly identifies that and says hey what you're looking for is not directly available however there is a way for you to get to that INF and that's what it sort of details back to me and towards the end you can see that it also tells me the significance of that I do the same thing with flash it took just 2.9 second this is impressive right and again because and if you see if you read through it it says a similar things which is it's not directly available you may want to sort of do or provide extra things and has a significance and that's what I was mentioning that every file that you upload you can actually just delete it right after you ask the question so it's not necessary that you have to sort of keep those files always in the cloud all right so the last thing is the video this is one of the challenging things and I have one video from the last year which is um Ai and the Google search not this year feel free to sort of load any other MP4 as well there's a little helper function that you would have to write which sort of extracts all the frames and then send those frames to the model so that it can sort of give you those reasoning I'm doing exactly the same thing I'm saying hey describe this video and how is Google using generative Sears give response and bullets and you can see the Simplicity of all of this I'm not even trying to do any kind of system instructions profiling or you know act like this act like that I'm just trying to be very very simple in time in in sort of trying to ask these questions and it's a deliberate thing because when the analyst will use a system like this they would not do all of these things like as a normal user they would write the way they think which is random combination of everything so I do the same thing again and you can see that again it sort of goes through the whole 1H hour video um sorry this is not a 1 hour video this is sort of 20 minutes uh video and it took roughly 8 to uh 8 to 9 seconds and again it sort of gave me all the summarizations and so and so forth I do the same thing in the Flash and it sort of took 5 Seconds slightly less uh but again it sort of came back with the results the last thing is you I can also get to the embedding part which is I can have a text and I can convert this uh into embeddings now I'm going to assume that a lot of you know what embeddings are um but at a high level just imagine that you're taking bunch of text converting into numbers and these numbers are going to help you then later find out similar things together that's all we need to know right now there's a whole Theory maths behind it I would say that you know you can go back and um sort of explore that more so I'm using the new model text embedding 004 which is one of our powerful models that we recently launched so now so far we good so going back to the presentation so we did the first part of the validation let's go back to the presentation and okay so as a developer I can say all right I'm happy um and I can make my boss happy because I can say hey the files that you gave um and the models that I have access to um I can pretty much sort of do most of the things that you asked me to and it is able to sort of go through each and every modality that I threw at it and I am getting the answers so I'm sort of pretty happy the boss is pretty happy and as with most things you know how uh the leadership sort of becomes greedy and they're like hey if you can do this why not sort of scale this up uh they're really happy because of the results that they're seeing so now you get the new requirements and slowly like I said we're moving into the lines of the production so the new requirements looks like this where the leadership says that hey um this is good but then if you want to do this in inside the financial domain we need to really do things sort of slightly different because there's a lot of security there's a lot of Integrations there's a bunch of things that you would have to do when you want to make things sort of more secure so they gave you these requirements which are it should have some kind of an access control not every analyst will have access to every kind of file you have some kind of Access Control uh that you should have access to it should be able to scale right now we doing this on 10 documents how about doing this on 100,000 documents you need some kind of a uh platform some some kind of a service which can help you scale that uh you also need a platform because again for most of you when you build Services it's not just one simple prompt and getting the answer there's a bunch of things and bunch of microservices that you need to write so you need to be in a platform where you can do all of these things very um very easily and then security of course that's one of the major things that you would always want to focus on but from a developer perspective again as a developer you'll be like I don't care about any of those things I want things to be easy um and that's where I don't want any of the work that I just did on Google AI jini API to go waste I want to use all of that which means I should have a very easy migration um I should have a storage I don't want to use Google Drive because my analyst would not save it there I need to have lot of operations and this is where mlops llm Ops and all of that comes into picture I should have ecosystem where I can do all of those things and of course the tuning part the tuning the rlf distillations bunch of other fancy things of course we don't have time today to go through all of these things but we'll try to sort of build our ideas around it and later we'll see that how we make ourselves sort of more Enterprise ready without sort of doing everything on day one we'll sort of try to be ready for that okay so let's jump into the demo do remember like I said the goal here is very simple we just want to reuse the code code that you just saw and we just want to see how easy it is for us to do so that when we are in the cloud side of the things things are sort of Enterprise ready by default again I'm going to give you the links this is where the notebooks are if you've not taken a photo feel free to take the photos uh this is where you see all the notebooks and the cookbooks if you want to sort of Explore More okay so I'm not going to go in detail of this notebook because it's exactly doing the same thing as the last notebook but I'm just going to show you the things that change when you go between two platforms um for whatever reason again you don't have to but if that's your choice if that's what you're trying to do you would want to remember the migration that is uh required so of course because you're moving between platforms you would change the SDK so now I'm sort of doing Google Cloud AI platform the authentication changes so previously when we were doing the API key this would not work similarly in the cloud in the cloud you need sort of your Cloud account you need project ID there's a lot of security layers in those things for obvious reasons because you're trying to sort of make sure that everything uh remains in your tenant nobody has access other than this so that's bunch of Step that you would have to do and that's what I'm doing here I do the authentication it uh it sort of gives me access to my tenant where my data should be so that I can do all the things and nobody else can do that I'm downloading the same data from drive just to make a point that we can actually use the drive things as as well as well as the uh cloud storage as well so I'm downloading the same thing importing would slightly change for obvious reason because you're doing that this step remains same except the name so if you now focus on the name part you will see that in the other side you had at the latest in this side you would want to sort of uh remember that the name slightly changed because now um in the cloud you have still preview because there are the slas are slightly different when you do this between cloud and the uh on the outside so here you will have a keyword preview and for the both models and again I have both the models here as well so both the models are available in both the ecosystem this code exactly Remains the Same no change whatsoever I run the same code same uh helper function just to show you that just by changing few things you can sort of just switch between the platforms all right good enough so now if you so go through all this step everything sort of Remains the Same I'm running the same flow just want to point out one quick thing so this is where you have uh an option you can still do and read things from uh Google drive but because I have access to cloud storage I can actually put things in cloud storage and read everything from there directly without the need for me to download anything in the central location or any server or anything I can just point the model to say hey here's the file which is where you see my GCS URI and I sayy here's my file just go and sort of give me what I whatever I'm looking forward to and that's one thing to keep in mind and that's one of the things that you know will come in very handy in the next part where we'll try to not download anything and do everything sort of in memory so that there's no footprint or anything of the files that we are playing with so you saw both the options so you have flexibility to either use open data if you wish to if you're like no no I want things to be secure you can sort of put it in the GCS bucket and call it a day rest everything remains same uh all the same quotes same prompts nothing changes same output and everything uh as is okay back to presentation because like I said we're moving towards production slowly so now we know that we have a good migration that we have done so we are in cloud and next we'll see why and what do we do in the cloud so just to do a quick recap the API when you move between these two different things there are only three things that needs to be changed or updated which is the library installation dependencies authentication if you just focus on that rest everything your helper functions your prompts your instructions everything remains same so you don't make that change and again I want to be very specific I'm not saying do that shift or migration you have option both the ways whatever you choose as a developer I like choices I don't want to be forced into a direction and that's where this slide comes in handy that if I ever want to make that sort of move I know where and what to do with this so this is good but let's talk about production right because again production means lot of things production can be so many different things to so many different people so many different companies I'm going to do something different in a sense that I'm going to create a flow or let's say just the extension of what we are trying to do so on the left is what we are doing right now and bear with me and you know we'll we'll sort of slowly understand what what's happening on the left side in the Prototype I had to choose a file and ask a question which means I need to know which file and what question to ask right like I would need to know that can we automate that can we make some kind of an intelligence layer where I don't need to know what file the answers are available I should just ask the question because now again think about it isn't that what analysts would want to do they're like I don't care where the information is I need to know what these things are you figure this out where the answer is available and that's exactly what we trying to solve in the production flow you can build that production flow in either of the ecosystems there's no uh this that oh only this will work here and not there you'll see that in the code itself but we are taking one well lit path we are saying okay now because we are in the cloud and we are trying to be Enterprise ready let's build that intelligence layer with the use of different models embeddings so that later I can just ask a question and it should a find those documents where the answer could be and then give me the answer answer so that's the kind of automations we are trying to do uh just a quick thing because lot of people do get confused as to why are we trying to be on the other side and like and this is where I said we are trying to be Enterprise ready we're not trying to develop everything on day one we're trying to be ready for the future requirements and what it means is essentially you need to have ecosystem components where bunch of different things are available to you so Vortex is one of those features in the cloud where you have different models you have collab collab Enterprise workbenches AI Studio developer sdks you have bunch of these and again uh the difference here essentially is that these are all managed Services right these are not Services where you would have to worry about a lot of things so this is almost like a software as a service all of this taken care so you can augment your things you can customize you can train you can use rlf bunch of these things are already built in there you just use do and uh call it a day and then you can also sort of do lot of orchestration mlops layer and bunch of those things and it doesn't sort of only stop there but there is a this is just one part of the cloud which is the focus geni part but the big aspect is the cloud has so many things and I'm not going to bore you because I'm not the sales guy here uh but there are a lot of things there are sort of uh compute storage networking you know all of these things right so let's get back to the demo side and this is one where you you'll have a lot of fun okay so before I do that I just want to quickly show you that uh just the collab that you're using there's also an Enterprise version of that where if for whatever reason again we trying to be secure right we are saying hey I don't want to use any of these things on the outside of the world you do have collab inside the cloud ecosystem as well and this is how it would look everything sort of remain same it works exactly the same just that it'll be inside your tenant what I've done is um and of course because we're doing produ ction we can't just use those 10 documents let's try to challenge ourselves right and to do that I actually took this challenge I said okay I'm not going to work on those 10 documents I'm actually going to work on more than um pretty much like 30 35 documents and then I created categories I said okay let's call categories like blog post earning calls if you open this I have bunch of earning calls I have bunch of podcasts so I have a dialog dispatch podcast converted all of that into the MP3 I have product launches vide so I have Gemini product launches videos there and the reports again for year-wise for 2023 quarterly reports I have annual reports for some years so you can see that I I really went out of the way to make this really complex right because but then this mimics like the production flow this mimics something that we would essentially try to do in the end which is you'll have your data sets like this and this is again inside the cloud storage if you notice so this is not sitting outside in the world or not in the Google Drive this is sort of sitting inside the cloud storage secure only accessible to specific people and just to show you these files this is the same uh PDF that I was showing you but this time this is sort of on the cloud I have videos now which are sort of um uh product launches and things like that I have audio which is earning calls and so and so forth let's try to build this thing and we'll go step by step my goal here is not to sort of confuse you but sort of go step by step I'm going to give you all these steps in a very simple form M um it sort of assumes a lot of things but it's okay we'll sort of go through it again The Notebook links if you want to but before we build that here's how things should look like by the end of that notebook if I put a query and let's say there's a black box if I Sol that a function called get answer I should be able to get that answer this is very different from what we were doing in the Prototype right because in that we were passing the file and then asking the question this I'm not even passing anything I'm just asking a question and letting it figure this out for me so that I can just get the answer that's what we're aiming for and it's quite easy like if once you get the hold of it what we trying to do it it's very easy simple design pattern okay so not boring you with all the authentication importing and everything I'm using two models I'm going to use both the models just to see how things change just to spice things up let's start with the first process there's a lot of pre-processing that we need to do in order to sort of um make this achievable the first step I do is I essentially go through each documents that I have if you go through the I'm not sort of also going through the helper functions that makes us do that um but if you actually go through you'll see that I'm not downloading anything I'm actually doing all of this in memory so I take the storage I take the blob I do all of this processing and then save it nothing outside of it so the simple process here is I go to each document which is a PDF I extract all the documents and the pages and then I come up with something like this which is like a metadata so I have a text type which is like whether it's a quarterly report which year is it I have a GCS path so this is not accessible outside I have page number and the text which is there so that's what the first step I'm doing and this is how that metadata looks like and you'll see how and why this becomes very important if you see the amount of files and the pages I'm processing ing these are the files that I have and it shows how many pages it has again we trying to push the boundaries of what we can do with these models we trying to see whether we can do that or not in the first place the total files I have are 32 um sorry I said 40 total files I think it's more than 50 um the total Pages are, 1600 and that's what we trying to challenge um and if you just see one page output you'll see that this is what one of the pages random index sort of looks like let's do the same thing with audio I take audio I send the audio to 1.5 and say hey can you Summarize each and everything from the audio I'm not trying to transcribe the audio at this point I'm just trying to say hey just tell me what this audio is all about just give me the high level summary and later you'll see why we are trying to do that once I do that and you can again see it's the same process I have a prompt that I've written so I'm now trying to sort of go a bit uh more in the prompting I'm trying to be very specific I'm giving it spef specific things saying summarize contextualize analyze synthesize I'm giving it specific things and again it goes through each of it and you can see it took 14 minutes on 18 files roughly less than a minute to process all of those MP3 files gave me the metadata and this is the output of it so you see it says thematic analysis of alphabet earning and then it goes into segments it goes into uh different different segments of where the audio who's the speaker because I've written this in my prompt I've said that you break this th uh break all the things into segments and then in the end you'll see there's a Q&A session conclusion uh prominent tees and overall purpose good I did this with all the audios the next step is obvious I'll do the same thing with video which is to say I'll write a prompt and I sort of again you can come up with your own prompt but I'm saying that do a concise summary thematic context connections and so and so forth again it goes through all the video files it took 13 minute and all of this I'm doing in Gemini 1.5 Pro I'm not using flash a because I I want to be very accurate I want to be uh and maybe accurate is not the right word but I want to be very comprehensive and sort of go through each and every layer of it which is again less than a minute for each video because I have 18 files and it took 13 minute to extract all those things again let me show you how the output looks like for one of those files Segment 1 segment 2 segment 3 synthesis again it's following those instructions in the prom that I've given okay so far so good so now I have these three doc uh three data frames one which has text uh which is extracted from the PDF second is the audio summary SEC uh the third is the video summary what's the next step so the next step is very simple um you can imagine that these are such a huge text in one go and because I'm trying to find similar things and we saw that the analysts are asking very strict uh sort of very specific question they're not trying to ask a whole story to you right so I want to chunk all of these things into very small segments and for an example I took 500 but this is what this is what it means you take the text you split all of these text individually and then split them further into chunks you can do a lot of Innovations here a lot of experiments I'm not trying to tell you this is the right way I'm just saying that I did in this example 500 character split which is to say just to show you an example imagine if this was the main text in the page it got split into three smaller components which is the ones that you see down this will help me because again remember the analysts are going to ask very specific questions I want it to find the information which is specific to it rather than going through the whole document because the specific information may be in those uh specific chunk okay so we're done with the chunking the next thing I do is I take each of these chunk and I send it to the text embeddings um again like I said all we need to know is these embeddings are important because with these embeddings you can then find similar things you can say hey this is what uh analysts have ask and these are all my chunks what are the chunks which are similar to it now I'm sure you would understand that because you can do that you can actually find which videos and audios are also going to have the answer which analyst is asking because we have converted everything into text we had the summaries of audios and videos that's one part of the problem and you can imagine this is very similar to the rag pattern again for those of you who know you know this is how rag works we'll do again one bit of um sort of complexity there and like I as I said like I used to call it spiciness because I'm trying to sort of make myself uh take that challenge so we'll do one change in the rack pattern which is I get the text embedding all good but let's try to see if you can rather than passing the text can I pass the actual file to Gemini 1.5 and we'll see that in a moment so now if you see the final data the three data that we have or the one that we are calling it as metadata we have embeddings we have GCS path we have text we have page number and again you would notice that I'm maintaining all of this because I need the citation I need to be very sure the analyst needs to be very sure because they can always go back and say oh this is saying from this document this page number let me sort of go there and check this that's why I'm maintaining all of this and all three data you can see there bunch of numbers as an embedding the last thing that I do and then we'll do the big review is the fact that I'm just building a simple index I don't want to search across three uh data frames it's going to look very stupid uh I also don't want to keep all the text and everything I just want embeddings and the index some kind of index so that I can go back and figure this out and this is that Vector DB kind of things right you've all heard Vector DB is making all the things you can store it in open source Vector DBS like choma DB you can use uh Cloud to do Vector search your choice whatever you want I'm using Panda's data frame which sounds stupid but at least it'll help us get to the point that we trying to do so that's my index this is the index that we going to use to find which files are going to have the answer which analyst is asking and for you to know which actual file is because if you see the actual index there's no GCS path so I can't go to the actual file but we'll write a simple logic because they're all built from the same file so each data frame that you saw also has the index so so you can do a simple one-on one mapping and get to the original file okay so far so good so I have everything um I have taken a backup of course this whole step from the start till the end takes a lot of time because you can imagine uh just after combining and chunking all of those Texs it took roughly an hour to get all of these embeddings and everything again that's a scaling question you can sort of figure this on your own on the uh other side of the ecosystem or you can sort of write Cloud Runner Cloud function and scale things up and make this really fast your choice let's do something interesting so I save this I'm loading this up all these files um and the next thing is I'm writing a simple now this is where the retrieval and the ranker logic comes into picture what this simply means is that if the analyst is asking a question I should be able to go to my index find where the information might be available take those files as is with whatever modalities they have and then send those file to Gemini 1.5 so that it can give me a much better answer remember this is a slight change from the rag pattern in rag you actually send the text right you actually send the summaries and everything I'm saying I just need enough information now you would realize why I was doing the summary and not the transcribe because all I need is one confirmation that the answer is available in this file as long as I know that I'm anyway going to send that whole file to Gemini that's the 1 million context benefit so if you see through these functions um one thing also that you'll notice see this um I am not even downloading anything I'm actually reading everything from buffer I read I save I extract those pages where the information is I create a separate PDF and that's what I send so I'm not sending the whole PDF I'm sending the PDF Pages where the answer is and then I create the instruction so this is my instruction that I'm sending to Gemini saying hey all of these files that you have can can you please answer the question and again just to show you that when I actually send the files I'm actually sending the GCS path because I'm not I don't have this anywhere local and again we said right this is important for us we don't want anything in the middle layer we want everything to be part of the GCS and then in the last I just do uh calling of the Gemini model okay so how will this work so once you write this thing let me sort of Zoom this up let's say I have a question question which is what is the role of AI in accelerating the progress of un sustainable development and I have two questions again like I said you now know this enough that I'm trying to sort of always push the boundaries so you have two questions and I call the function that I just walk you through again you have the access to the code go see it's a very simple code nothing complex see what I'm passing I'm passing the query that the user asks I'm passing the index that we created those 100,000 things this is where you can also call Vector DBS if you wish to and I'm passing the model and I'm saying just give me the top five files where the answer is so before I go into the answer let me show you what essentially um those citation looks like so the question that you ask by the index and the simple cosign similarity with the summary it said hey the answer might be available in these Buns of audio files and the video files and that's what we're doing here we are saying these five files where the answer is available I'm going to send to the model which is G 1.5 so if you see the The Prompt that we are s sending to Gemini just have a observation so I write the task I mentioned the questions so you can see those two questions see the contextual files again I'm not loading anything up I'm not sending the summaries of the text or the way we do in rag right we send all the context of whatever information is and this is different because now I'm sending the file itself I'm saying hey I don't know where the answer is I know that it's available I now know this I'm just sending you the whole file the whole audio and the video and see how many files you're sending I'm sending uh four audio and one video that itself is like 5 6 hours of content and asking the same question which is the question that I had it goes and answers that very well so now if you see the answer it took 54 second to do all the process to find those five files went to Gemini gave those five files and then gave me the answer that I'm looking forward to that's what we said that's what we want to achieve it's very simple it's it's not even complex like I maybe I may sound like it something complex I've done but it's very simple if you go through the code you'll get the idea so again because the kind of uh prom that I have written you will see that it it does question it does answer and then towards the end it gives me each and every attribution of the citation where this is and that's pretty much it um if you see I've given many more examples so this is another one where I'm saying how does Gemini long context work and if you actually see it again gives me the answer if you check the citation there are video all the video files have this answer the next is I'm asking another question I don't know where the answer is but it figured out that it's available in text again it gives you the answer so that's the bit of how would you sort of at high level do all of this and I hope this is interesting to you and that is all that I want to show let's get back to the slides thank you oh