BigQuery vector search and embedding generation

Key Takeaways

the rapid growth of unstructured data like text images and video offers hidden insights however it Shear volume and complexity pose a challenge in extracting its value traditional keyword searches struggle leaving important information in connections undiscovered this is where Vector search offers a breakthrough Vector search uses embeddings which are numerical representations of your data machine learning models transform text images and even video into embeddings allowing you to compare and search them based on their underlying meaning not just literal keywords this is because semantically similar text images and video have similar numerical representations in embeddings Imagine finding images similar to a product photo identifying video clips related to a customer review or finding text documents that all Express a similar sentiment ctor search makes this possible and what if you could do all of this without leaving your data warehouse well big query makes it a reality in this video we'll explore big queries functionality that enables you to generate multimodal embeddings and perform Vector search let's start by understanding how big query handles not only structured data but all types of data including text images audio and video big query is a powerful and flexible tool for bringing together all of your data assets using object tables big query lets you directly connect to and analyze unstructured objects in Cloud object stores you can combine insights from your unstructured data with your structured tables and unlock new analysis opportunities to understand complex data like free form text or images you need a standard way to represent them and embeddings make this possible machine learning models take your unstructured inputs like images and transform them into numerical vectors these Vector embeddings capture the underlying meaning of your data making it possible to compare and search based on semantic relationships not just surface level keywords big query machine learning lets you generate embeddings for your unstructured data directly from your data warehouse using familiar SQL syntax it also supports multiple embedding models to suit your use case you might choose use a text embedding model for all of your textual data or a multimodal embedding model for semantic search across modalities like text image and video you simply invoke the ml. generate embedding function over an object table and embeddings are returned directly to bigquery with your embeddings stored in big query you can index them and use the vector search function to find data based on semantic similarity this means that regardless of the input modality you return relevant results based on the actual meaning big query Vector search is optimized for analytical use cases and can efficiently process large quantities of data and handles the infrastructure management for you previously these types of analyses required complex pipelines and external tools which created silos and friction big query brings it all together embedding generation Vector search and your existing data enhancing your analysis experience let's head over to the cloud console and see how this works imagine I work at an e-commerce company in the Fashion retail space I'm thinking of introducing a new sweater to our collection but first I want to check whether we have produced or sold anything similar I have sales data in big query tables and I also have a vast catalog of product images in cloud storage but unfortunately none of these images are tagged or indexed we can see a sample image from our catalog here in cloud storage I'm going to then navigate over to bigquery I begin by creating a cloud resource connection this allows big query to access our images in cloud storage and the embedding models in vertex AI to add a connection I'll click add connection to external source connection type as vertex AI remote model and then I'll give it the name myor conon and click create connection I can then navigate to the cloud resource connection and copy the service account ID I'll need to give the service account a few more permissions so I'll open up I am and another tab I'll assign the service account to roles storage object viewer and vertex AI user then I'll move back to Big query with permissions all taken care of I'll now create a remote model so that I can access the embedding model through bigquery the create model statement takes a few inputs a model name a remote connection which I just defined and an endpoint in this case I'll use both text and images as inputs so I'll specify the multimodal embedding model this remote model setup is one time for the embedding model and I can continue to use it in this project the last setup step to generate embeddings is to create an object table this will give me a structured interface to my object in cloud storage for this example the object table will display metadata around the product catalog images object tables require a few inputs these include a table name a cloud resource connection and the Google Cloud Storage URI where the product catalog images reside note that object tables Only Store metadata and that there's no movement from cloud storage to another location with our setup in place I can now use the ml that generate embedding function the function takes two inputs the first is a remote embedding model that I defined that takes multimodal inputs the second is an object table that references objects in cloud storage then I'll run the query behind the scenes Big query is submitting all of the images in the object table to the multimodal embeddings API hosted in vertex AI it's then returning the results and storing them in a big query table once the query finishes running I can check out the new table here we see a number of fields the first is a field containing a vector embedding the next field displays the status in case of an error and the function also returns all of the fields in the object table associated with each image so instead of spending time outside of big query to generate these embeddings and load them in later I've instead generated everything without leaving our data warehouse with embeddings in a big query table and ready to use let's now turn to Vector search I'll begin by creating a vector index on top of my table this will enable faster and more scalable semantic search by using an approximate nearest neighbor search algorithm I need to give an index name specify the big query table it refers to and Define the embeddings column within the table lastly I can input a number of options in this example example I'll specify that the index uses cosine distance and you can find other options available in the documentation big query will create a vector index for me and I can immediately run Vector searches but full indexing may take some time depending on the size of the embeddings table I can check that progress using the information scheme of views notee that Vector indexes are fully managed by big query and are automatically refreshed when the indexed table changes now I can run that sem IC search the following query does two things it first takes a text input and uses the ml. generate embedding function to generate an embedding using our remote multimodal model it then takes the embedding and Compares it to all of the indexed embeddings I've specified one optional argument so that I only return the top five results again I'll be searching my product image catalog of clothing images to see whether we've ever sold similar products in the past I'll input the string comfy rainbow sweater and check out the results I return the top five matches and the cloud storage Uris now I'll run the same query using a big query Studio notebook and a helper function that renders the images and it looks like we've sold similar sweaters in the past for my next steps I might want to look at the sales data for these sweaters to see which were the best sellers or maybe look at how many were returned turn to us and for what reason I might also consider summarizing the customer reviews for these items to get some inputs for our new product there's a wide array of possibilities on how to proceed with further analysis traditionally working with embeddings and Vector search require deep knowledge of specialized libraries managing complex pipelines and overseeing additional infrastructure big quy dramatically simplifies this process with embedding generation and Vector search built in and with a familiar power of SQL you can unlock new insights without needing to leave your data warehouse with embedding generation in Vector search and big query you can expand the scope of your analysis examples include building content-based recommendation systems content classification and outlier detection and even retrieval augmented generation with big query's Lang chain integration give it a try check out the description below for a step-by-step guide and resources to get you started