Google Gemini Embedding 2 Tutorial | Multimodal Image Matching Project

Key Takeaways

Hello and welcome back to the channel. In this video, we are going to build a really interesting hands-on project using Gemini embedding 2, Google's first natively multimodel embedding model which was released recently. Traditionally, embedding systems are mostly focused on text. So, if you wanted to build a semantic search over text, text embeddings worked really well. But the moment you want to work across multiple data types like images, audio, documents or videos, things become more complicated. You often need separate pipelines. Maybe one model for text, another for images, another process for audio and then some custom logic to connect everything together. That setup works, but it becomes messy very quickly. This is where Gemini embedding 2 becomes interesting. In case you interested in learning Genai, we have a platform where you can learn it for free. You find the link in the description or you can click the I button here. Google introduced it as a natively multimodel embedding model which means different kinds of content can now be mapped into same vector space. That is a big shift because now instead of saying this is my text pipeline, this is my image pipeline and this is my document retrieval pipeline. You can start thinking in a much more unified way. A piece of text, an image, a PDF, an audio clip or even video segment all can be presented as embeddings in a shared semantic space. And once that happens, that kind of system you build becomes much more powerful. For example, you could search images using text, retrieve a document from a screenshot, match audio directly without transcripts, build multimodel rack systems, or search across mix enterprise data where information exists in many different formats. So this model is not just about embeddings getting better. It's about embeddings becoming more general purpose across modalities. Now let's quickly talk about what this model supports. Gemini embedding to work across multiple modalities including text, images, videos, audios, and PDF documents. It also supports interled multimodal inputs, which means you can combine things like image plus text in one request. That's useful because sometimes meaning comes not from one modality alone, but from the relationship between them. Another nice feature here is flexible output dimensionality. The default embedding size is large and informationrich. But the model also allows smaller dimensional outputs. That matters because in real world system there is always a trade-off between retrieval quality, storage cost and search speed. So depending on your use case, you can choose a higher dimensional embedding for better quality or a smaller one if you want a lighter and faster system. Now let's talk about the project that we will build. We will be building an image matching system. So the idea is simple. We will have a small data set of images belonging to different people and then given a new query image, our system will figure out which person that query image matches the best. Now what make this exciting is that we are not going to train a CNN from scratch. We are not going to fine-tune a classification model and we are not even going to create a heavy computer vision pipeline. Instead, we will use Gemini embedding 2 as semantic feature extractor. Generate embeddings for images, compare them using cosign similarity and use that to retrieve the closest matches. And while the project itself is an image base, it can also give us gliss into something much bigger. Because Gemini embedding is not just an image embedding model. It's a multimodal embedding model, which means it can represent text, images, videos, audios, and documents inside a single shared embedding space. Let's get started. In our project, we will use 3072 dimensional embedding which gives us a strong semantic representation of image matching. Before jumping into the hands-on, let's now understand the project we are going to build. Inside the data set directory, I have separate folders for different people. Let's now understand the project we are going to build. Inside the data set directory, I have separate folders for different people. So, the structure looks something like this. One folder for Nitka, one folder for Janvi and one folder for Wasu. And each folder contain multiple images of that person. Our objective is simple. We want to read all images from the data set. Generate an embedding for every image using Gemini embedding 2. Store those embeddings locally. Take a new query image. Generate its embedding. Compare it against all the store embeddings using cosign similarity and retrieve the top matches and then predict which person the image belongs to. So conceptually this is a retrieval problem first and then a lightweight classification layer on the top. It shows that with modern embeddings you can build something meaningful without building a full deep learning training pipeline from scratch. No model training, no complicated feature engineering, no annotation heavy process. Embedding model does the heavy lifting for us. So the first step is loading the API key and creating the Gemini client. So in the notebook I have loaded the API key from the environment file and initialize the client. This is the basic setup that allow us to call Gemini embedding 2 from Python. Once that is ready, we can start sending data to the model. And next we have done some configurations like giving the data set directory path giving the model name which is Gemini embedding to preview using the embedding dimension 3072 and giving the part of the cache file so that we don't have to create embeddings again and again until we change the data set and next we will initialize the Gemini client. Now I have created some helper function which take inputs like JPG, JPEG, PNG, WEBP which take the path and find the files, check their data types, normalize the vector, find the cosine similarity, show the images and finally creating the embeddings using Gemini embedding 2. So now let's load our data set. So I have created a function which take the data set directory path mark label as the folder name and map those images inside the folders with the label. So we found total images 151 labels found were Janvi, Nitika and Wasu. All the three folders we have created. And now final step let's build the embeddings. So I have created a function which take the path of the data set. Take the path of the cache file where our embeddings will be stored. And here I have given the parameter force rebuild as false. So once we give the path our embeddings will be created and here are the embeddings. So after running the cell our embeddings will be created and there are total 151 embeddings which is equal to the number of images we have given. So now to find the best matches I have created a function which take the part of the query image and then return stop by similar results. So first it will take the part of the query image and create embeddings and then find the cosign similarity between all the images and the query images and give the result to us. And now the function predict person and this function will return the name of the p. So we are done with the setup. Now let's try out our image matching system. I have given my query image path here and it will return the top two most relevant result to us. So let's try it out. So here's the query image and this image was not present in our data set and the results are rank one score is 0.8 which is correct because both of them are the same person and the next image rank two is also correct. Now let's try out on similar images of two different people. So I will provide a query image which is kind of similar to the image which is present in our data set and it will return the top six relevant images. So here's the image of Janvi and in the first rank we got this one which is the same image which is 100% match but if you look at the image at rank two it's of Nikka and the image given was of Janvi but we got a high score because the pose was same and both share same facial features. At rank three, we got another image of Nitika with the same pose. But if you see at rank four and rank five, we got the images of Janvi again. So this system is not perfect yet. But in the future with few developments, we can make it more better. And that's it for this video. I hope this gave you a clear understanding of how Gemini embedding works in practice and how you can use it to build a real image matching project. If you found this helpful, do share your thoughts. And if you want, you can extend this exact notebook into a larger multimodel retrieval system. In the next version of this project, you could even connect it to a vector database, add threshold-based unknown detection, or bring in text and document matching as well. Thanks for watching, and I'll see you in the next video.