Create advanced data driven Gemini API apps

Have you ever tried to build a complete rag system? Well, rag is hard. In this video, I'll show you how to build powerful rag systems fast using Gemini's file search API. Plus, I'll show you some advanced data processing features if you hang around. I'm Mark McDonald. I'm part of the Google DeepMind developer experience team and I work mainly on Gemini. This video doesn't require any coding experience, but it's made for developers who are building AI apps. So, having some experience doing so, even by coding, will be useful. Let's go. If you've ever built an app, you know that data is important. You have content, something in your app that is important to you and your users. Maybe it's your personal to-do app. Maybe it's an analysis engine that you've built. When you build an AI-powered app with something like the Gemini API, you learn that AI isn't magical in and of itself. The real magic comes from combining your special sauce, like your to-do list, with focused AI prompts that work together. Enabling your users to say something like, "What am I missing from my to-do list?" or "Which file has the important thing that I forgot?" allows you to leverage your unique value proposition to create magical experiences. However, LLMs are finite. They have context windows that impose an upper bound on the amount of data you can process in a single request. And even if they were infinite, more tokens means slower responses and higher costs. Having the ability to use a million tokens is pretty powerful, but sometimes less is more. Rag, or retrieval augmented generation, is the usual way to scale beyond a model's context window. This video won't explain it in depth, but essentially, you put all of your content in a data store, then when you need to answer a query, you selectively include relevant chunks into your prompt so that you're only working with relevant info. This sounds simple, but the truth is that rag can be pretty complex. There are different flavors of rag from regular naive rag to agentic rag, self rag, graph rag, iterative rag. At the architectural level, you already have a number of choices to think about before you even start designing your system. Then, each component of the rag system needs to be considered, too. Which vector DB to use? Will it work with my own environment? How can I host it? Then, you need to think about how to optimize your document processing, things like chunk size and overlapping windows, plus how you'll unpack complex file types like PDFs with tables and other unstructured data. And then, your retrieval step has complexity, too. How do you map your user's question or prompt to search the data store? Do you set up query expansion or document re-ranking? There's a lot to consider. And in a world where we want answers fast and AI promises rapid results, this can really slow down your prototyping. This is where the Gemini API's file search tool comes in. File search is a fully managed solution for rag that is built into the Gemini API, abstracting most of the indexing and retrieval stages away so you can focus on your core logic. It's powered by the Gemini embedding model, a state-of-the-art embedding model based on the same architecture as the powerful language model that you know and love. This ensures that the text isn't just matched on similarity, but on understanding, too. And it has automatic OCR built in so you can index multimodal content like PDFs, but with a purely text index. Similar to a regular rag pipeline, there are two main phases. Ingestion, that you do ahead of time to get your data indexed, and then search. Search is even easier here than a traditional rag setup, as it's just a tool that you tell the Gemini API about. Then, the Gemini model will use the tool to find the relevant documents, add them to the context, and produce whatever output is needed to fulfill the prompt. One of the powerful benefits of having file search provided as a tool is that the model is able to repeatedly call the tool, inspect the results, and refine or update the search queries that it is using to find the right information. We call this agentic rag, and it's one of the most effective rag engines currently known. In this example, a user has sent a query asking how to apply for leave. Within that single generate content call that your app makes, Gemini will repeatedly make tool calls using the file search tool to answer the question. This is the agentic part. The user's query is ambiguous, as there are different types of leave that could apply. So, instead of stopping and asking the user, Gemini starts with a query to find all of the different types of leave processes that might apply. Then, having read the results of those searches, it knows that the user needs to fill out specific forms, so it searches for them next. Then, since the process requires approval, an additional search is done to figure out that step, too. And finally, once the process is understood, well documented within the model's context window, it can formulate a response to the user detailing the different forms and processes required for whatever type of leave they are requesting. As you can see, the model can orchestrate a lot of processing as the result of just a single generate content call from your app. So, it's still important that you have a human take a look at any output before you use especially if you are familiar with the Gemini SDK already. Here, we initialize everything and create a new file store. Then, upload your files one by one. Note that here, you just upload them directly. There's no PDF preprocessors to define or chunking strategies. You can do any preprocessing you want if you want, but we have some smart defaults set, so most content typically works without any configuration. Once you've populated your file store, it's ready to search. Searching is built into the existing generate content API, so searching is just a regular prompt with a file search tool attached. Here's a sneak peek at an app we're going to expand on later. Let's look at the ingestion code first. You can see here we create a file store if it doesn't already exist. And here we upload some local documents into the store. Now we run it and watch it all upload. Okay, let's go back to the code again. Here we have another tool to query the file store. Just like the snippets I showed you earlier, this tool provides a regular Gemini API chat interface that has the file search tool attached. You can see we have our file store present. Now let's run it to make sure our document is uploaded. There it is. Since this is an example, I have it showing the full JSON response. You can see you get quite a bit of additional context here. As you saw in that example, the API will generate a grounded response based on the data that you have in your file store. If you were watching closely, you might have seen that we got more than just the model's response. Also included some citations. This is one of the built-in benefits of the file search API. It'll generate a response with links back to the grounding chunks it used, so that you can provide your users with links back to specific documents, and for supported formats like PDF, even specific pages within the documents. Sometimes you or the user will know that there are certain files that should be included or excluded when generating. An example of this in our book searching app could be if the user selects that they only want to search a specific author or for books in a given time period. Data like this may be tricky for a model to be able to identify from the text alone since a book doesn't write the author and publication year on every paragraph. Fortunately, this is a simple enough fix with the file search API. When uploading your content, you can specify arbitrary metadata that is saved along with the text itself. Then, when you are generating responses, you can specify any filters in the tool specification. When the model searches the file store, it will first apply the filter you provided so that the Gemini model only retrieves candidate chunks that are relevant. And when you get back your grounded citations, you'll also see metadata. This could be helpful if you need to display or highlight any particular metadata, for example, grouping the citations by author or sorting by year. You may have already seen that the Gemini API can produce output that adheres to a given schema. It's helpful if you need a specific schema or just some specific fields, but you don't want to parse a text response. Well, you can use the file search API together with structured outputs. This means that in a single API call, you can pass a prompt with a complex request and have the Gemini API identify all of the relevant information from your index, analyze the results, and generate structured outputs to answer the query. All right. Now, let's look at how to build an app like this from scratch. We're going to use Google's Anti-Gravity IDE and start with a new empty project. I've created a new project called BookNook, and you can see here that there are no files in it yet. The first step when working with the Gemini API is to install the Gemini API dev agent skill. Agent skills are little text snippets that you can install in your coding agent that explain how to do specific tasks. We're adding one for the Gemini API to ensure that the model always has the latest, most up-to-date information about what the Gemini API can do and how to do it. This skill connects Antigravity to the Gemini docs and ensures that it always reads them when working with the Gemini API. The link on the screen will take you to the instructions or follow along here. Open up the terminal window and run NPX skills add to add the Gemini API skills. Select the Gemini API dev skill and then ensure that it's going to install it for Antigravity. For this project, we're just going to install it into our project, but installing it globally will ensure that it's available for all of your future projects, too. Now, back in the main Antigravity view, you should see the skill listed in the file browser now. Now, we type in our prompt to get the indexing part of the app built. This will generate a script that creates a file search store and uploads files to it. This is the ingestion part of the rag pipeline. In this example, we're using books from the Gutenberg project, which contains a huge archive of literature that's in the public domain. Okay. Now, let's try it out. We can see that it's importing the books we've saved. Now, we add a UI for searching our books. This will create a website for us to engage in a chat conversation with our entire library. Let's try it out with a query. Let's see if we can find any characters who are famous for their athletic ability. Voilà. That was an intro to building your own rag systems with the Gemini file search API. Before we go, let's take a look at three more new features we've added for handling data when you're building with the Gemini API. This will be a quick fire round, so lock in. If you already have data in Google Cloud Storage and need to get it into your Gemini API prompts, you can now pre-approve your GCS buckets for use with Gemini and then pass the bucket URI in your prompt. No more uploading your files every time you need to use a file. Similarly, if you have content with a different cloud storage provider, you can now use that directly with the Gemini API, too. Sign your URLs when you upload them to your provider and pass the signed URL in your prompt, and now Gemini can use that content, too. Again, this saves you having to upload multiple times to different places. When you're building real production apps with the Gemini API, you'll have some API calls that are super high priority and others that can wait. Now, by specifying the service tier in your requests, you can either spend a bit more to request higher priority traffic or mark your traffic as flex so that it could be delayed if needed, lowering your costs. You as a developer will know when you're making an API call that is serving a real user or an important use case, so the request needs to be top priority, or alternatively, if it's for background or offline work that can easily wait a few more minutes for a response. Now, you can pass that information onto the Gemini API and use it to optimize the cost and experience. Okay, now you've seen how to build powerful database solutions with the Gemini API's file search system. Plus, you've learned how to bring in data from cloud storage solutions like GCS and S3. You know how to optimize requests for lower error rates or lower costs. Drop any questions you have into the comments or just share what you plan on building with these features. I'm keen to hear what you're all building. I've been Mark McDonald. Happy building. >> Mhm.