Gemini Embedding 2 Is a Big Deal

Key Takeaways

Okay, so Google just released Gemini embedding 2 and it's a big deal because this is the first unified multimodal embedding model from Google. That means the same embedding model can process text, image, video, audio and documents into the same unified space. Now Google had embedding models before and even multimodal models but traditionally every modality was treated differently. So for example, audio would be transformed to text using speechto text model and then we would use an embedding model to process that audio. Same goes for images and PDF files. We'll have some sort of transformation in between. Now that transformation results in a lot of information lost. But with a unified embedding model, this model can have a universal semantic intent and understanding of the original modality. A good example of this is audio. If you do speech to text and then text embedding, you basically are capturing the words, but you lose the tone, urgency, and background context. But if a model can directly process it and encode it, it is going to preserve the semantic intent. Now, Google is not the first one to create these multimodel models. I previously covered call. This is a multimodel model that traits images and text using the same vector, the same embedding model without need for this intermediate transformation. But Google has probably taken this to the next level. And if you are building a retrieval system, you should definitely look into this model. Now, it's state-of-the-art on a number of key benchmarks, but I'm not going to spend too much time on the benchmarks. I want to show you some practical examples of how to build with this model because I think this unlocks some really interesting use cases for retrieval augmented generation systems. And one of them is that you can combine different modalities and also during retrieval time you will be able to retrieve different types of modalities. Now in the rest of the video we're going to divide it into two different parts. In the first part I'll show you a practical example of how to get started with this. And in the later part I want to focus on how to think about building full stack applications which goes beyond simple retrieval. Now under the hood it uses the Matrosushka representation learning which means from the same embedding model you can use different dimensions based on your needs. This is going to give you a good compromise between cost, accuracy and speed. It has support for over 100 different languages which means you can build multimodel retrieval systems. But more importantly these embedding models are not just for rag or retrieval augmented generation. You can do sentiment analysis, document classification, and clustering with these. Okay, before showing you a practical example, some really important details. First, just like LLM, embedding models have their own max number of tokens it can process. This can go up to 8,000 tokens for text that transfers to about six images per request for images. It can support up to 120 seconds of video input. The audio is going to be natively processed. They haven't really given how long of a audio it can process in a single go. But for documents, it basically converts every documents in document page into an image and then processes that image. Okay. So now next, let me walk you through how to get started with this. I'll show you how you can use this to build a search engine and then I want to also talk about a full stack application that uses this new embedding model for more advanced retrieval. Okay, so I put together this example notebook. Link to this is going to be in the video description. You'll need a Gemini API key. I am downloading some example images from Wikipedia. This is basically a multimodel search engine. Here we have text description. It's a dictionary. There are specific tags that we're going to use later. And the idea would be to take these audio narrations, create audio files for those, and then embed text, audio, and images in the same vector space. Here are some example images that I have downloaded. Here is a helper function that is going to take different modalities and compute embeddings for each one of them. We can do retrieval based on a combination of different modalities. So this first example is using text for search or retrieval. We want to look at ancient ruins in the mountains. You can see that it correctly identified this beautiful scenic image in Peru. There's another query is talking about snorkeling in warm tropical water. And this is the image with the greatest semantics similarity. Right? So using the same embedding model, we are able to embed text and images. Now traditionally you would pass the images through a clip model that would create text representation but you don't need that intermediate step anymore. Now you can use the same setup to do image based search as well. So here we have this input image and the model is able to look for very similar images. Then we can do same thing with audiobased search. So here's an audio segment and we just provide that audio segment which says I want to visit a place with beautiful coral reefs and tropical fishes for snorkeling. And since it's a cross modality search it's able to pick up this nice image of coral reef. Right now this notebook provides a lot more information. So here I'm doing cross modality similarity. So I highly recommend to look at this. This will give you a really good understanding of how these different modalities are present or embedded in the same vector space and they keep the semantic similarity which is pretty awesome. Now in the rest of the video I want to focus on this application that I'm building and this is basically an agentic search system that is going to also use this new embedding API. So to show you an example right now I'm embedding 20 different documents. We have some audio file uh some PDF files. Some of them are related to an acquisition. The other ones are academic papers. Right now we can use this model to do clustering. So I ran the clustering mechanism. Uh this uses the scikitlearn clustering. So not directly that embedding model but we could do something better and that is classification of documents. So I took the same embedding and then we ran those through an analyzer which will basically classify them into different clusters and on top of those we pass this through an LLM and this does automatic tagging of your uh documents for you. So you can use this model for a lot more than simple information retrieval systems but it still can be used for rag or retrieal augmented generation. So I'll show you a couple of examples. In this case, I'm asking for financial adjustments described in the closing documents. So it's able to retrieve those documents for me. And if you look at the sources, you're going to see there is a PDF file and then an audio file. Now in this case, I'm trying to build this as a whole SAS application. You can even see I'm keeping track of the API calls by different users. And there is even an option for the user to upgrade to a different plan. Now all of that is powered by charge beam who are making this video possible through their sponsorship. This is a unified platform you where you can build subscription plans for your users and even track API based usage. More on them later in the video. Okay. So if you have seen some of my recent videos, I talked about this agentic file search where you can combine traditional retrieval augmented generation system as a first filtering layer and then use agentic file search to get a much better comprehensive responses from your retrieval system. Now I have covered this in some of my previous videos, but let me show you the architecture of this new application that I'm building. And this basically combines this new Gemini embedding 2 model for multimodel retrieval with aentic file search. Now when we upload any type of modality then there is modality specific chunking mechanism. So for example for PDF we talked about that it can process six pages by converting them into images. For audio, we divide this into 60-second segments. And for text, instead of 8,000 tokens, we limit it to 6,000 tokens. And each of one of these have overlaps built in. Then everything is processed through the same embedding model. The embeddings are stored in duct DB. Then for clustering, right now I'm using a simple K means clustering algorithm, which can definitely be improved. For every cluster, we generate labels using the Gemini flash model and the results are stored in the duck DB database. Similarly, we also generate different categories for every document. This helps with agentic file search and the raw files are also stored in the database. Now during a retrieval time when the user query comes in, we look at the embedding based semantic search. This is going to get us the most relevant chunks that we think from the vector DB. Then as the next step we look at the source documents for every each of those chunks those are passed on and to the Gemini model. Now in the process we also look at in the source document if there are cross references specific to that user query we also include those cross reference documents when we are sending it to the LLM to generate final responses and this gives the agent full context of whatever the user is asking about not just based on simple few chunks. Okay. So in terms of the overall architecture here's how it looks like. You have the front- end client. Then we use Firebase for authentication and for storing all the different databases. This include chat history. Then we use the wheat dev proxy. For API calls, we're using Gemini API both for embeddings as well as for the LLM. Now on the backend server side we have duct tb for storing the multimodal embeddings and then we have chargeb billing. Now this is critical because it not only tracks the usage per user it also has information about different subscription plans what different entitlements are available to those subscription plans. I'll show you in a minute to how to set this up. But it can also track API calls to Gemini API. So you probably have seen some services offer different amount of credits based on the plan that you have. You can actually build something like that using charge B. If you want to learn more about the Aentic file search project, I have a number of different dedicated videos on it. The code is open source. Link is going to be in the video description. Now in this last part, I want to actually talk about how this whole thing is set up. So let's say if you ask something like what is the purchase price then initially you can see that it looks at one specific document but then it realizes that it refers to some other documents as well. It goes does cross references look at all those documents and give us the final answer. Now this system makes a lot of API calls. So I want to limit the number of API calls specific user can do. So right now there are only 50 API calls for pre user free users and if somebody wants to move to another plan they can click on upgrade to plan. Here is the pro plan. They can provide their billing information and they will be able to check out. All of this is powered by charge B. Now let me show you how to integrate this into your own apps. So right now I have test app. This is the dashboard that you're going to see after signing up and creating your first project. Now, in order to create different plans, you will need to first create a product. So, if you go to the product family, right now there are three different product families. If you want to create another one, just go provide your product family name, description, and product family ID. In my case, it's hybrid rag. This is the ID that I'm going to be using. Now with every product you can have associated plans. So right now I have two different plans. The free plan and the pro plan. If you want to create a new plan, you can just provide those details here. Okay. So we're going to look at the pro plan as an example. So this has,000 API calls. The free plan has 50 API calls. Also I'm setting up max file size, right? And these are the things that you can control through entitlements. So if you go to entitlements, you're going to see features. So I created a Gemini API calls feature which is connected to two different plans at the moment and you can actually see full logs. So for example I can go to events and I think in the last 12 hours I had four different events. They're logged in here and these are the different API calls that were made by Gemini or through the Gemini API. Chargebby is an excellent platform to build not only subscription plans but enable usage based billing and you can track everything within the same platform. Okay. Now, in order to integrate this into your own applications, I highly recommend to look at the API documentation. But here's the best part. You can just point your agent to their new skills for charge fee and the agent will be able to implement everything for you. That's exactly what I did with my application. Anyways, do check them out. Link is going to be in the video description. I actually highly recommend to check out this new embedding model because I think this is going to enable some really interesting applications. But as I said, semantic or embedding based retrieval is not enough. You want to look at beyond that and you want to treat this as a tool in a list of different tools that are available to you. Now, in order to see what is possible with this, they have a number of different demos. I'll put a link to some of those. Now, the model itself is still in preview, so it's not production ready yet. Also, it's relatively more expensive compared to its previous iteration and some of the other offerings. But do check it out. I think it's very interesting model to keep in mind if you're building retrieval or classification or clustering systems. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next