Build AI agents for e-commerce with ADK + Vector Search

Key Takeaways

Hi everyone, welcome to another episode of Hands-On with AI agents. And in this episode, we'll be building a rag agent using Google's ADK and vector [Music] search. In the first part of the video, we'll be saying what vector search is and what are the different kinds of embeddings. And in the second part, we'll be diving deep into building the agent as a rag agent. And to do that, we have Cass here. Hi, Cass. Hi s hi I'm K de developer advocate from the crowd AI team I focus on the building demos and blog post and documents for their vector search embedings and ADK lovely let's get started then so first let's think about the typical design of rag or retrieval augmented generation systems for the e-commerce chat bots here we have the user query can you find the pixel uh Google pixel 9 then LLM receives this query and decide it needs more informations to provide an accurate answer and suppress harsh issues. So it uses the the retrieval back end like a vector search and uh uh get the result from the for the query and then generator the answer to the user. That's the typical the rack system setup. Now uh let's look at some challenges for the usual rack systems specifically multimodel search and keyword search. This example shows a user asking can you find cups with dancing figures and the SKU or product number is one to three ABC receives this query and thinks how do I found the images and keyword search. This query is more complex than the previous one because the it involves the descriptive informations like cups with dancing videos which implies the visual features rather than the text semantics and also the specific identifier or product name like one to three ABC. This represents a common challenges for the rag system in the e-commerce site that uses a simple text sim similarity search. They need to be able to understand and effectively pro process different types of the information in a single query. Another challenge is recommendations. Here we see a user asking can you suggest the birthday present for my son. The LM receives this and thinking how do I make a recommendations to the user's query. These questions demonstrates the complexity of the recommendations. It's not just about finding a similar items to the text query, but understanding what might be the suitable suggestion for their sum. This highlights the another area where the simple retrieval text similarity search doesn't work and we need something more sophisticated and provide meaningful recommendations. So to solve those problems, we'd like to dive deeper into the advanced practices for the vector search that go beyond the uh the simple similarity search. Here are the some best practices for the higher search quality. Specifically, I'd like to discuss about multimodel search, hybrid search, and task type embedding. This is how multimodel search works. So with multimodel search uh you would use the multimodel models to generate the embeddings. Those multimodel embeddings is shared across the m much modity uh right the images and text they share uh the same the embedding space. That means if they have the similar meanings that result in a closer embedding distances. This enables the text to image search or image to text searches. And to do so, you can just use the vertx AI embeddings API to use the multimodel uh models to generate those multimodel embedings. Here's another of the solutions uh we'd like to use which is the hybrid search that uses the the keyword search and the semantic search with the single the the search index. So the problem of the using the simple simulator search is that it is limited to the what the embeddings model understands. It struggles with the product names or newly added product names. This limitation is could be a major issue in a production ra systems. Hybrid search addresses this by combining semantic search and keyword search or so-called sparse embeddings into a single vector search index allowing a single query to retrieve the best mix of the results. Okay, I'd like to share the the actual example of the margin model search with this demo. This demo has the three medium items provided from numeric.com which is the popular e-commerce website uh with the built with the march model uh embeddings and index. So for example if you type the queries like cups with dancing figures then from the 3 million items you can instantly find those cups. Please note that these the items are found only by looking at those images. So this demo doesn't look at the any text title or categories or the text descriptions at all. The embeddings model understand what's going on in those images and find the the items that has similar meanings to the text query cups with danc figures. Now let's take a look at the another example which is the hybrid search. In this demo, I'd like to use the semantic search index combined with the spruce embedings for the keyword search. So from the 3 million items, you can find things like 1 2 3 4 that doesn't have any meanings. But you can do the vector search with that with this first ambings keyword search. So that you can get the items like you know these items where you have the 1 2 3 4 as the keyword in the product description or the titles. So you can combine the both the result from the keyword search and the semantic search uh in a single result. So we have covered the two topics in the advanced topics for this vector search. Now I'd like to discuss about the the third one task type MX. Why we would need this? The problem is that in many cases the simple similarity search doesn't work for the production systems well because the in most cases the query and embeddings has the different semantics like this one. Why is this skyable this query the answer scattering of the air they actually have the quite different semantics or meaning as sentences. So in last 10 years uh the many researchers in the information retal area has been struggling to to solve this problem by using the the machine running and deep learning models. The popular solution here is the so-called dual encoder or tutotor model as you're looking at the animations that has the two part for learning the different domains like a query domain and a database or document domains. The thing is this tutorial model runs the relationship between the different query and documents so that you can get the best relevant result rather than the similar items found on the documents database. But you don't have to hire your own teams with the data scientist to build your own tutorial model and train it by yourself. Rather than that, you can just use the Vert.xi CI embeddings API to get the task type endings embedings that is generated from the pre-trained tutorial models we provide. So with that task type embedings you can ask the questions like why is the sky blue then that will be that will generate the embed that has the closer distance with the answers like a scattering of the air. Okay let's take a look at the actual uh the demonstration of the task type embeddings. First I'd like to use the semantic similarity uh search with the query like birthday present for my son. Then if you are using the usual semantic search then you'll be getting the result like this. So you are looking at the the many key chains because they have the very similar uh the text descriptions like the uh the sun and birthday but maybe they are not the the ideal items you want to get as the result. So instead we are switching the model embedding model to the question and answering task type embeddings and by using this with the same query and the same items the result will be quite different because the task type emitting models runs the relationship between the query and the relevant items. In this case, these are the present items. Also, this the result has the totally different semantics like a regy mouse. Uh but the models can recommend the result like this. Yeah, it's nice to see that the results are not all keychains, right? But now that we've seen all of these different types of embeddings, how do we take this and integrate with our AI agent? Yes. So let's dive deeper into the how you can take the advantage of the advanced practices of vector search combined with the AI agents. So let's think about another challenges for the e-commerce website that is smart recommendations. So we have already discussed about the recommendations by using the task type embeddings. But sometimes user would ask like this can you suggest birthday present for my son? what's your latest trends? Then the LM receives request and wonders how do I make a smart recommendation like a consult. These questions requires more than just finding a specific items. It demands an understanding the context like what's the latest trend on on um recommending the items for the birthday present and making a personalized suggestions. This highlights the need for the more sophisticated approach for handling personalized and trend aware recommendations. The solution here is combining the AI agents with the vector search practices. The UI agent takes the user query and triggers the Google search to research the rate strength then generate a bunch of queries for finding the interesting items and asking search agent to search items. The search agent acknowledges this and running a 20 queries in power. Then started generating related more specific queries like STEM toys for 10 or science kids, science kids for 10 and experiments experiments for kids. These queries are then passed to the vector search which can provide a much broader and more relevant set of results. This match agent approach allows a more intelligent multif faceted search strategy going beyond simple keyword matching to provide a pretty helpful [Music] recommendations. Okay, I'd like to show the actual demonstration of the combination of the AI agent and vector search that is called showers con. With this demo, you can ask any the ambiguous or vague questions like birthday president for 10 years old some. Then now the AI agent uses the uh Then now the AI agent uses the vector search and issues a couple of queries to get the result. You can take a look at what's going on. uh under the foot by looking at the console. You can see there are three queries like toys and games and action figures are generated and run against the 10 million items to find those the result you are seeing. But now you can ask a deep research to the agent deep research. With this deep research mode, the agent now uses Google search to make a research uh on what's what kind of the items people are buying for this kind of the query like the person present for 10 years old. Then with that it defines the five different item categories and issues 20 different quer queries for each five uh item categories and get the result like this. And under the foot you can see on the console that it generates the bunch of degrees total 100 queries for single deep research request. So rather than having the users typing their own queries on the uh the search the box uh you can use the AI agent to make a smart recommendations based on the research result from the Google search and picks the best result from those 100 queries like this. Those are all diversified uh interesting inspiring result. You can see this is this goes far beyond from the usual recommendation systems you would use uh with the e-commerce website and also because the AI agent has a capability of the understanding images. You can make a query by using your own images. For example, you can upload your own images for the view your room. Then the AI agent understands what's going on here. It's some homework setups and uh find the some suggested items like this. So it's not just a simple similarity search for the images. It's it's about understanding the context and intent and find the relevant items like this one LED desk. Then the agent even can generate the possible images uh when you're placing these items to the desk. Now that's a really cool looking UI cast. Now tell us how do we implement this agent that we just saw. Thank you. Yeah, let's dive deeper into how you can build this by yourself. So I have used the two technologies. One is the agent development kit or ADK and Vert.xi vector search product from Google. So what is ADK? ADK is the new opensource framework developed by Google and announced in April of this year. This is an open-source multi agent framework by Google. It supports of course Gemini the as the Raj model but it also supports the third party models and also it supports the RI audio and image streaming. So the demo you have seen actually Z is capable of to have derived the audio voice communication with the user. Let's examine how the deep research mode is built with ADK and vector search. When the user UI agent takes the user request, it uses the Google search for the grounding uh that learns what's going on for finding the items for the birthday present. uh by looking at the the current trends in the internet. Then the UI agent ask the search agent to generate 20 queries per item category. The agent repeats the this for the five categories total 100 queries for single search request. All queries uses multimodel embeddings, task type embeddings and keyword embeddings. The practice I have discussed earlier. These 10 queries are sent to the vector search in parallel. So you get the much faster result. Then with the result for the 100 queries, the search agent performs much model item curation. It reviews the item images and descriptions one by one. Just you know sharing the all the result with the user directly. The agent selects the items actually relevant to the user's intent and item category. So usually those the 100 or 2200 items you got with the search are filtered to under 50 items. Gemini curates item by analyzing images and user intent. The item images are passed to the Gemini as a curation prompt like this one. So this the the image tiles are sent to the Gemini and Gemini actually look at the actual the item images one by one and select most valuable items for the user. Okay, let's take a look at what kind of the code you would write to build something like this conception demo by yourself. This is a published uh notebook sample. So you can take a look at in detail if you're interested. So I'll skip those uh details for now and just start with installing the ADK. So to get started with ADK, it's super simple. You can just pip install Google ADK and that's it. And you may want to uh import those required libraries at first. Also you have to set those the environment variables to get started with the ADK by specifying the project ids and locations and so on. So before diving our agent uh before diving into the details of our agent uh we'll be defining an test functions for the agent. uh usually agents will be running in a runtime environment like a agent agent engine product but uh for this notebooks we define a simple runtime called test agent function uh I don't discuss in details about what is the runtime for the agents but uh take a look at the agent runtime documentations for details so here's the our the first definition of the our shop agent It's just a simple the basic agent with the Gemini 2.0 flash model without any external tools or such capability at this times. So let's test it by using a test agent runtime. What kind of the site is this? Then agent would respond I am a shop agent for an e-commerce site with millions of items. So this is the foundation of the our demonstration. Now we'll be adding the vector search capability key item search capability to the agent to do so. But first uh we'd like to define and call vector search function to call the vector search back end. The actually this is just an an HTTP request uh function uh as we'll be using the existing public vector search back end. So we don't actually going through uh building the vector search index and endpoint and so on. We just uh make a HTTP request to the existing back end. It takes this function takes the URL of the the uh endpoint and the actual query from the user and how many rows we want to return to the uh the agent. So let's define that. And now we will wrap the uh the call vector search function with an ADK tool named function find shopping items to. So this is a just wrapper function for the uh the the previous function. But thing is by defining the the proper function names and the signatures like the parameters and the doc string. So the agents can take a look at those the function signatures like a function name, parameter names and dock strings to understand the functionality each tool can provide. So you have to be very clear uh on on writing the those doc string like this one define shopping items for the e-commerce site and that takes the arguments which is queries the list of the queries and this function returns a dict of the following uh one property status or the items uh which is list of the items from the easite. So by defining this kind of the tool function the agent can easily take uh use that understand that and use that uh on the fly. So let's try this tool by passing the two sample queries coups with dancing people and dancing animals and then it issues the actual queries the vector search back end get and get the result like this. So now it's ready to extend our shop agent with the search capability. And here we have added an one paragraph to the instruction to find items use find shopping items store by passing the list of queries and answer to the user is the items name and description and image URL. And also another things we have added is the tools parameter when creating the agent object by passing the find item function. That's it. So that now the agent uh is aware of the existing of the find items tool and use that uh when it it is required. Let's try the agent with the cups with dancing figures. Then the agent thinks oh to solve this problem uh you have to use the find shopping items tool and call it and passes the result to the user like this. Now we will be adding one more agent called the research agent that is a a market researcher agent using the Google search on generating a query based on the search result from the Google search. Let's take a look at the actual instructions we passed to this agent. When you received a search query search request from a user, use Google search tool to research on what kind of the items people are purchasing for the user's intent then generate five queries finding those items on the e-commerce site and return them. So this is the role of this agent and uh one big benefit you can get with ADK is that to use Google search as the grounding source you can just refers to the Google search tool. This is a built-in tools. So you don't have to define it by yourself. Uh and and so you can just write this line and import the Google search like this. And that's it. You can easily access to the Google search as the grounding source. Let's test this. So for the query like birthday present for 10 years old boy it now made a query with the Google search and generate some generated five queries like this stem kits sports equipment building sets outdoor gear and games. Now with those all capabilities we can finalize the shop agent. We defined the instruction like this. First the agent should do the market research using the research agent as we defined previously. So that the the agent research agent will generate the five queries and share those generated query with the user first and ask if they want to continue with the search. The second step is to find items using the find shopping items tool. So by passing those five queries to the this fine shopping items tool you can get the result from the vector search back end. And as you can see in this demonstrations we have used the research agent as a tool. This is a design pattern so-called agent as a tool. So rather than having the two agent as a peer mount agent system we use you we are using the sub agent as a tool. So the uh the main agent the shop agent takes all the control over the multi agent systems. So let's test the the final shop agent with these all capabilities. So first you when you ask with the query like can you find a birthday present then it uses the research agent to generate those queries by using Google search. the those generated queries are like a stem kits, Lego regets and remote control cars and it ask you do you want me to search for for items using these queries. So you will say yes to that. Then the agent uh uses the fine find shopping items tool to use the vector search back end to get the final result as you see. So uh with this demonstrations we have shown a so-called generative recommendation capability that uses the uh not only the vector search pack and it also uses the uh external tool such as the Google search tool to extend the queries uh based on the users intent to get more uh the relevant and interesting and fun result from the e-commerce website. That was a great walk through Cass. Thank you so much. Okay. I have a one question. Can you tell why we using uh the agent as a tool instead of a sub agent here? Like why why would you opt for this design pattern instead? Yeah, this could be an interesting discussion. It should be defined by the uh what kind of the user experience experience you want to provide. For this case, I really wanted to have the single agent to be just like the bringing a role of the cons for the user. Usually the human conscious would be you know looking at the uh the books or the maybe they can be using Google search to get the result and summarize it to reply to the user and just like that I have chosen to use the the tool as a agent design uh to have the UI agent to be represented and understand everything but it's depending on the the your requirement for the user experiments you can also use your sub agent or peer agent design pattern so that you can pass request to the another agent and another agent can be on front end for the users as well. So it's totally depending on what kind of requirement you have. That makes sense. Thank you. So let's recap what we have done for building the shop conscious demo. There are three things we have done. First the using ADK uh for building a AI agent with the multimodel multi aent capability that has the realtime multimodel communication capability. The second generative recommendations because each AI agent has the intelligence by using the Gemini. You can let the agent to use the Google search for grounding to make a research on the latest strength and generate a bunch of queries and do the much model of item curations by looking at the item images and description. And finally those agent uses the vector search back end that has the everything practices I have discussed earlier like multimodel search hybrid search task type embedings and I also used the ranking API. So that was the ideas and concepts uh that is used for the postcon demo. Now that was lovely Cass why don't you give us more information and where to find more resources. Yeah to get started with those technologies. There are two uh resources you can take a look at. First one is the vector search documentations that is uh you can find on the crowd.google.com. There are a bunch of the getting started tutorials and notebooks and the blog post use cases. So please take a look at and another one is the ADK documentations and samples. Again there's another uh bunch of the getting started the docu documentations and samples and everything and also I have put the link of the my own samples uh for the the shoers conscious demo. So please take a look at on the description of the video. Thank you guys. Thank you everyone for watching this episode. We hope you really like this and let us know in the comments what you think about this video and what we should be building next. Thank you. Thank you. [Applause] [Music]