Weaviate TECH Hands-On: Query Agent in JavaScript

Key Takeaways

[Music] If nothing I will kick this off. We're about 6 minutes past. So today we have about 53 minutes and some change. The idea is to have a really hands-on session diving into the query agent showing how it works in different scenarios. If you're unfamiliar with the query agent, the query agent is a a gentic tool that we launched to make queries much easier. And so a lot of what we'll be doing today is in JavaScript by the way. I am Daniel Pir. I'll be helping facilitate this. I'm a developer experience engineer um on the engineering team at W8. And hopefully I can help you out. So like I mentioned about 52 minutes we'll go through the we query agent some live coding and then a bit of question and answer. So is everyone familiar with JavaScript? No worries if not um the query agent is also available in Typescript. So if you'd want to take a look at what the query en query agent rather enables, we can quickly go through this blog post. So we create query agent. So introducing the query agent and so essentially this is something to to help run queries uh across multiple collections in an agentic manner. So, this is a little thing that we put together. Um, and I will share a link to this in case you want to take a read. Um, but I'll actually explain it um in my slides. So, uh you might know V8 for being your go-to database for things like semantic search, keyword retrieval, or hybrid retrieval, and maybe even rag. And so the idea behind the query agent was to make all of this much easier by introducing uh an agentic workflow to how you interact with WV8. A lot of times our interactions with WV8 were one to one i.e. we make a specific query uh and we're always querying a specific collection or a specific tenant. Um, and for agentic platforms that simply did not scale because there are some use cases where you have multiple collections and you want to be able to access a lot of collections or at least you don't want to have to write or hardcode this collection logic in your applications. And so the benefit of the query agent is it builds a brain and this brain here is a large language model. And what that is, it exposes us to two tools. The first tool is a search tool and the second is an aggregate tool. And so essentially the large language model i.e. the query agent decides whether or not we should run a search or an aggregate or both over one or multiple collections. So whenever you interact with the query agent you can read data from multiple collections and the agent decide the agent rather decides what tool is best to read for those collections. So we enter natural language queries like what shoes can I buy for over $60 for example and the agent translates that query into a wevate query decides if that natural language query i.e. What shoes can I buy for over $60 is more of a search or more of an aggregate. In our case, this would be a search. Then this will look into what collections might actually be able to supply us the answer to that question. Look in that question, analyze the response, and if it's valid, send it back to our user. And so it does a lot of the heavy lifting for us in terms of building our rag pipelines. And so like was mentioned earlier, if you're looking to go from a naive rag where you simply pass all the context you get from a hybrid search for example to your large language model to then generate something. This sort of takes it a step further and upgrades that to an agentic rag where an agent decides what collections or what sort of data is useful for your application and for your users's query. So the idea like I mentioned behind the WV8 query agent is to essentially have a large language model run or rather make decisions for us with respect to what searches we should run, what queries we should be making and what tools we should be using. And so if you're looking to think about or rather if you're trying to figure out why you should use something like query agent, if you're tired of having to think about something like that or building an agent or an agentic application, this is the perfect thing to use for something like that. So, like I mentioned, based off whatever query you put into your um based off whatever query you put into your agent, it decides what collections to access. At initialization, you give your agent access to multiple collections. And so, based off of that configuration, it decides what collections you can access. It also determines if we need to run a semantic search, an aggregation or both. Um, whatever is best for helping the user find out whatever type of data they're looking for. And then it also formulates its queries for the best best performance in mind. I.e. It looks at whether I should include filters, whe what the right operator is, what the right keyword is based off whether it's a semantic search or a keyword search or a hybrid search because the search queries for those things are actually quite different. Uh, and etc. Uh, and like I mentioned, it also determines what filters are right for each query. And then based off whatever answers it gets, it formulates an answer and sends that back to the user. Another really important distinction is looking at the difference between a search and an aggregate. A lot of times when we make a search, we're looking for uh specific results that we then send back. Um and these tend to be more rooted in natural language whereas aggregate queries tend to tend rather to be rooted in arithmetic. So, if I'm asking something like, "Hey, what are um what are the top five brands um or what brand has the most amount of sales, for example, or like what shoe brand has the most amount of sales?" That would be more of an aggregate because we'd look over multiple items and see and sort of sum up all all of their sales to then see which one has the most sales versus a search um is more like what shoe can I get for under $70. Um so that's just to help you understand uh the difference between the two. And what we'll be live coding today is a bit of an application. We'll be working with JavaScript like I mentioned before. And essentially our JavaScript application will be interacting with the query agent. We'll be using search and aggregates as the tools exposed the query agent and then accessing uh e-commerce uh and uh brand um collections. So really important to actually use the query agent. We start by installing the agents package on npm. So the package is available in npm if you want to try it out. Right now it's in alpha which really just means that the API design is not set in stone. We will possibly be changing this design and that's why we mention it being in alpha. So there's just sharing that if you want to take a look alongside the agents package, you want to have the WVA JavaScript client installed. So I'll share this. And so once we have those two things installed, we can then run and use the query agent. Essentially what the query agent does is it piggybacks off of Essentially what the query agent does is piggybacks off of the client initialization and that we make with the WV8 client and so when you install this we agents package you pass in a client um object and that client object is what's used to interact with the agent. So let's get straight into our live coding. And so we'll make a couple of assumptions. Um before we get started, I'll assume that I already have data running. So let's uh run code. That's great. So now that this is open, we'll create a query agent of our own. And so what I'm working with here, and I'll zoom in, is the recipes repository. If you're new to WV8, new to WV8 building in JavaScript, this is one of the best resources that you can use. It is a repository of multiple recipes uh which essentially are endto-end scripts that help you run different or test out rather different features um and tools that work with VV8 covering things from similarity search to multimodality migration different inter integrations with tools like llama index um we'll be adding things like mastra and lang chain very soon. Hybrid search, generative search, and of course agents. And so this is where our code will be. We'll be working with the agents foldout. So if we get time, I might work on integrating the query agent into um a llama index agentic workflow. But the bare minimum for today will be using the WV8 query agent on its own in isolation. So I will run through this load.ts file. Let me know if this is too big for you and I will uh sort of zoom out. And so the first thing that we want to do uh if we're using the query agent is to get data into wave. That's one of the most important things. you can't query something when you don't have any data. Uh, also just as I as I keep going, if you have any questions, please feel free to to drop them in the in the main chat and I'll try and get them as I uh go along. So, I'll try and keep this as as basic as possible. So essentially here we have a import and so essentially we're importing a wevate client because we want to put data into we have a main function and this main function initializes a connection to weate cloud and in this connection we pass our or credentials because we're using the WV8 cloud and then call a function called populate we now populate V8 is where things get interesting. We initialize three collections. Uh if you're new to EVA, a collection is essentially like a table in a relational database. And so what we're doing is we're deleting any collections if they exist only if excuse me only if um we have the overwrite existing flag as true. At the moment it's false. Uh essentially we're doing this because if you try and create a collection when it already exists we it will send you an error and we are going to create uh some collections. So here we have our creation object and so when using the weate query agent there's a couple things in mind you have to pay attention to. The first being how you define your collection. If you're already familiar with how to define a collection, that's fine. This really mostly what I'm about to share really is mostly an extension of what you already know. And so if you're doing anything with agents, they benefit from as much information as possible. And so that's what you see here. We describe or rather add descriptions to both our collections and the properties. So I'll quickly just go over what we're doing here. So we're creating a collection called e-commerce. We describe this collection as a data set that lists clothing items, their brands, prices, and more. We create a vectorzer. uh essentially this is an embedding model that creates a vector representation of any data that you feed into it. So we have text to equivate um text to equivate and essentially we're creating a vector representation of the description of a clothing item and then we're also creating a separate vector representation for the name, description and brand. So once we do that, we then describe the various um properties in this data set. Before we dive into these, I want to actually look at the data that we're working with. So we're using hugging face data sets and we have an open data set. If you would like to play around with this, all of this is linked uh but I'll just share this in the chat as well. So essentially we have a bunch of different data sets. The one we looked at was the brands data set uh or rather the e-commerce one. So we have the e-commerce data set and so you can see we have a name, description, collection, a URL um and some sort of review of that as well as a vector representation. So you don't necessarily have to create that representation at least when you're working with this data set. Right. Besides that tangent, let's get back to explaining our collection creation. So essentially what we do is then describe each of the properties in this collection. And you can see that for category we add a description. Description has a description. Tags sorry has a description. And essentially these are all nuggets that the agent can use to decide what the best tool property collection is to help serve you best. So things like price here, the price of clothing in USD. Uh the fact that we include the currency USD is also something that the agent can look at when it's thinking through what the best solution is. So that's the first collection we create. We also create a collection called daily weather called weather like we did previously. We create a description or rather define a description and we give our temperature a description as well. So temperature value in degrees Celsius. Great. So essentially we do the same excuse me for financial contracts and then essentially run this load data set which downloads all the data from hugging face and puts it into weate. Now we'll explore my wave because we already have something. So just so we can see that we actually are already working with uh we will copy this code and run a query so we can see that we actually already have data in we so let's create a new file. So I'll just call this fetch.ts. I'll paste our imports. I don't need this. I will um close close that and I'll define three variables. So collection dot use. So we defined e-commerce, we defined a financial contracts. Financial contracts and we defined something called what was the third one? A third thing that we created was where is that? Yeah, weather. Great. I can just copy that. I think that makes more sense. So essentially what we're doing here is defining a variable that we can use to make a query. And so we'll say const uh e-commerce I'll use abbreviation ECC for e-commerce collection uh response is equal to uh our e-commerce collection dot query dot fetch objects. So essentially what I'll do is I will limit this to the first 10 objects and first three. This is just so we can see that we actually have objects and console log that data and got to await this. So let's run this in our terminal. Zoom in a bit. and we'll run npxtsx. And we're working in the agents folder. We're working with VV8 agents and the query agent. And we want to run fetch.ts. So you can actually see um what objects we're working with. Oops. Looks like I did not call the main function. Um so void main and I'll just describe our console node here objects fetched right that's why nothing is output and if we actually want to see something we have to specify objects so just so we can see that we actually have data so here you can see our properties um we have a skirt. We have a coat and we have what else do we have? Is this under name? And we have a software denim denim sorry. So essentially yes we have data in our collections and that's the only reason why I run that. Um so you can see that after running this you will have data in your collections. Now after we've populated data um what we want to do is essentially um start making queries. So this file run.ts is where we'll make our queries. We have a main function as we did before. Uh and essentially this query um agent is one of the uh classes that we use to build out our uh use of the WVA agent query agent. Like I mentioned, we always run this alongside the WV8 uh client. So we like we always do initialize a connection to our WV8 cloud and then initialize a agent query agent with a specific configuration. And so for e-commerce specifically we're saying new query agent and then we pass in a client object and the collections that we want to uh to work with. So we could absolutely get rid of multiple collections. So we could get rid of for example this e-commerce application and that should work perfectly. I have a couple queries here already I run and let's uh let's just comment everything below here. Does not work. Yeah, let's comment everything below here and type out our own query. Um, and let's it's con weather query. So let's actually fetch data and see what we have in our weather collection. So weather collection response and see what type of data and then we can start to figure out um right. So we have wind speed, visibility, pressure, temperature and these are some of the properties that we have. So we can start to ask things like I think one of the things that this agent is good at is exploring your data. So we could say something like con uh weather whether response is equal to this QA config. So query agent with config. We're passing the financial contracts and the weather and we'll pass in our weather query and choose the collections that we want to interact with. Let's get rid of what is the weather in New York and um say what is the average. Let's look at our data again. So temperature wind speed. So we could say something like what was the lowest temperature in May for example in 2023? What was the lowest temperature in 2023? temperature. There we go. Um, and then this weather response we want to display. So, we have a very nice helper function called display which will log uh our response. And when we get that response, you'll see how interesting this agent can be. So, let's go run this. I just clear my screen for a bit and type run.ts. So it takes quite some time to run. Um I think about five to 10 seconds because there's lots of back and forth with queries. Um right. So here we can see and we can see what our response is and this is a great opportunity to explore the response object from our query. So one of the things that we get back is the original query. So this is what the user put in. And so we entered what was the lowest temperature in 2023 and we get the collection that was used when this query was made any searches that were made and then any aggregates that happened. So previously when I shared slides I sort of talked about the difference between aggregations and searches where a lot of aggregations tend to be rooted in arithmetic and um numerical calculations versus searches being more um keyword based and rooted in text. So you can see we ran an aggregate on the weather collection um on the property temperature and we looked for the minimum as a metric. So the minimum value of all the temperature values is what we looked for or rather what the agent decided was the best thing to use it. Um and essentially it ran a filter because I did specify what was the lowest temperature in 2023. It also included a filter that added the first day of 2023 to the last day of 2023. And so you can see all of that information is included in that aggregation. And then for usage and I guess statistics get to see how many requests were made, how many tokens were used up in that request, how many tokens were used in the response, how many tokens in general were used as well as how much time it took for that answer to come the answer from the actual aggregation and then the actual answer generated from a large language model. So these are all things that we have and you can see here our sources is quite blank. So if we go back and change what our query is from what is the lowest temperature in May 2023. We could um could sort of say what day had. So we could say will it rain and so this is something you can you can you can see where the will it rain in May 2023. Did it rain because that's the past. Let's run that. And so there's this assumption that the model understands that precipitation equals rain. Um, interesting error I have not seen before. Will it rain in May 2023? Let's try that again. Oh, in all of my very thorough testing, I have not come across this error. So, that is good to see. Um, I'll just take note quickly of what I typed and give this as feedback. Um, whether connection. Okay. And seems to not like this. So, what we'll do is we'll get back to our collection here and we'll add our e-commerce. Our e-commerce collection. I believe this is how it was spelled. E-commerce. Yes. And we change our weather query and just change this to query response. So query response just right. So query and we'll change this to our normal query. And this query says I like vintage clothes and nice shoes. recommend sum of each below $60. So let's run this. So you can see the search functionality that is enabled in the query. Oh, it looks like there is a problem with the client. I will just test something really quickly. Ah yes, sorry. Before I do that, I actually made a mistake on the collection I was exposing. So, I've actually added the e-commerce client. Just take a look at something really quickly. Okay, that's very weird. Let me see if there are any pushes made. That might be okay. There's definitely some changes, but let's try and read through this error that we're getting. Error while searching we collection details collection name e-commerce. Right. Looks like we have a problem with this collection name. Um maybe we need to specify. Oh yes, it's because we did not specify what target vector uh we were searching through. So when I run this now, we should get it working as expected. Okay, we have uh a new error which means we're making progress. No target vectors were provided. Right. So in my query I need to specify what collection I want to interact with. if I define that collection. But if I get rid of that e-commerce, it should use what I defined in my configuration. Um, and then that should work. So, we got some uh troubleshooting for free here, right? So, here we go. Now we have our output and like we saw previously we have our original vector or rather our original query which was I like vintage clothes and nice shoes recommend sum of each below $60. It shows what collection name we searched through. And unlike the previous query which was more of a unlike the previous query which was more of an aggregation, this is actually a natural language query. And so we can see that we had our search query as vintage clothes under $60. And then here we run a couple filters uh and our query was nice shoes on this collection. And we looked for a property name. We looked rather for category and used the like operator of shoes and nice as well as um a description and a name of shoes and nice. So we're basically searching for anything that says nice or shoes on either the name of an object or a description or the category. And so the query agent essentially creates all of these um filters which can help us find in a much better way exactly what we're looking for. We get more information here on um exactly what we're doing. So the usage no aggregations were run and importantly here we start to see something that we haven't seen in we haven't seen before besides the total time we have a property called has search answer which is true and then we have a property called is partial answer which essentially tells us if the agent did not have enough information to confidently answer our question. That is set to true. And then we have an array of missing information which we set to specific recommendations for shoes under $60. So, while it does return something, it doesn't necessarily have all the information it needs to give us a confident answer on recommendations for nice shoes under $60. And then, of course, we have our final answer and we include the sources that helped um sort of inform this final answer. And so, that's one really interesting part of the query agent. Or uh you could also run a uh an aggregate query and see essentially what is the name of the brand that lists the most shoes. So it will look over all the brand shoe brands and look for the brand that has uh shoes and it reports that to us. So it says there is no available information on the brand of the list of most shoes in the provided data which is also behavior we want if we don't necessarily find something in the data we're looking for that's fine right so we'll comment this out and skip to this concept of follow up queries. And so essentially one really cool thing that the agent can do is follow up answers. So previous responses that we got and then run a similar search. So I'll comment this out and I already have a example here of where this is used. So we have a very basic query that we're saying or basically asking the agent saying I like vintage clothes and nice shoes. Could you please recommend some of each below $60? which was the previous query that we ran and this was the answer we got. We're displaying that and then making a follow-up query and passing in the context from the previous response as information for the agent to consider as it services this query. So, I like the vintage clothes options. Can you do the same again but or anything above $200? So, let's run this and then we'll see the differences um in the responses that the agent uh gives us. So, it should take about 10 seconds or so. Yep. 7 seconds. Right. I forgot to save. Okay, let's try that again. So, first things first, we already see that it gives us a response for vintage clothes under $60. But then you see our follow-up request here is can you give me anything above uh $200? Um, so it tells you also uh the missing information which was the variety of vintage clothing options above uh $200 beyond the single item bound. So there's only one item found of um $200. And we could actually test this with an aggregate query. Um, how many items? How many vintage clothing items? How many cost more than $200? So, we'll use that as a followup. And I think we're almost out of time. So I'll start to wrap up pretty quickly. So you can see we got that result. All the vintage clothes under 60. And then so we get an error here on the ids provided just covering one of so aggregation error right so essentially we're not necessarily providing the amount. So let's look at the e-commerce collection and all the properties in it. So we can actually properly inform. So run fetch objects on e-commerce and this just goes to show that you have to price. So, it would be nice if we gave this access to the price as well. So, maybe the price could be useful. The Yeah, I think price is the only other thing that we we need. Okay. So let's run our previous query again. And now that we've saved that, our agent should have access to the price of objects and then would be able to answer our question here. Right? So we get that answer for the first query and moment of truth. Boom. There we go. So, this just goes to show how important that configuration is both at query time and at configuration time. Our agent can only deal with whatever data we give it. And so because it didn't have any access to the price, it wasn't able to run an aggregate on the cost of something. When we gave it access to that price, it was able to see that we actually had items over uh $200. So there we go. Um which also sort of um invalidates our previous previous response when the agent didn't have access to the the price of something. Uh but at the moment this is what you can do with the query agent. You can build this into an API into a front-end application. I feel like it's pretty powerful and will help sort of take your rag implementations um to hopefully another level. So let's look at what's coming. We just had a release which is why I was checking GitHub on the agents. So we have um better types and we able to stream method rather we'll be able to stream data from the agent and we fix some of the typing progress and a couple other things. So keep an eye on this agent's npm page. You should see a couple releases coming up. And of course, as always, you have access to the recipes folder if you are eager to try that out. We'll be constantly adding new recipes to both the Llama index agents as well as the WVA agents. I'll also share the agents documentation page in the chat in case you're curious and would like to read up on the theory behind it or reference it. That's pretty much it for uh this. We have about eight minutes. I just want to share that next week Tuesday so essentially a week from now we'll be running another another workshop an online workshop on agentic rag workflows with weave in JavaScript I'll be hosting that session on zoom um and we have a lot more workshops coming up so if you would like to keep track of that this is the perfect place to do that you can find me in the company Slack as well as the community slack and online as Gambas. Thank you so much for attending.