AI Agent for Big Data Analytics | Amazon Web Services

Key Takeaways

Hello. In the next few minutes, you will learn how to use AI agents to analyze big data using open source trans SDK in AWS. I'm Chandra Ready. I'm a senior manager for Genai specialist solution architects at AWS. So what we will do is we will build an AI agent to explore big data and we will also have the AI agent generate business ideas and run natural language queries against the big data. So the text tag that we're going to use in AWS is uh we'll use strands agents and tools MCP servers Amazon bedrock anthropic cloud sonnet model AWS glue and Amazon Athena. Our goal is to allow users to generate research reports based on the data that is stored in Amazon S3. They know they have park files in Amazon S3 but they need to generate research reports. So how do they do that? They will build an AI agent that is going to use AWS Glue to crawl the data, catalog the data, um, and create essentially the metadata for this data and then it's going to use Amazon Athena to query this big data and then use the large language model to respond back uh, and generate research reports. The data that we're going to use is New York City taxi and limosine commission trip record data. Once you download the data and upload to S3, this is how the data would look like. This is are all these are all parket files. So the task ahead of us is that let's build an agent that will actually explore this data. Um so how many what is the file type? What's the count of file? What's the size of all the files? We'll identify ask the AI agent to identify five different industry verticals that will benefit with this data. and how do you monetize um the data uh in those industries and what are the limitations of the data and then it will create a nice report for us to read and of course we will also look at what are the token costs to the large language models to generate all these five research reports. So essentially it comes down to actually um system prompt prompt and really three lines of code to generate these research reports. The system prompt is where we are we specify the agent what what skills does it have what's the style and tone that it should generate the reports and the prompt essentially we are saying what's the task that it needs to accomplish um and this is where we are asking all these questions what's the type of the file uh how large are those files identify five businesses and generate the insights write SQL statements and what are the shortcomings and so on and in the mode what we are doing is we are creating a bedrock model and specifying anthropic cloud 3.7 sonnet to be used. We are specifying the temperature. We're creating a strands agents object by passing in the model the system prompt and we're also passing two tools uh which is use AWS and file write and we will double click on this in a uh in a minute and once we create the object we are passing the task to the agent and asking it to execute and once the agent executes let's look at what it has done what's the data structure that it has identified and uh the research reports that it has created Let's head over to the notebook. This is the data structure that it has identified. Um, as you can see here, uh, the table name, the column names, the type of the columns and the, uh, information in the columns. Again, remember we did not write any piece of code uh to create PIP data pipelines. Um the AI agent actually did it on behalf of us by talking to AWS Glue and creating this catalog and write uh and creating these data types and then it used Amazon Athena to generate uh queries and execute the queries uh and create this research report based on the data that was there in the park files. What you see here is that it was smart enough to figure out what who is the target audience and here especially we are looking at urban planning as smart city solutions. It figured out the target audience and uh as an example here it's trying to identify critical congestion corridors for that it wrote a SQL query and it found out that location 132 as an example appears in nine of the 15 most congested high volume routes. It continues to do its analysis, wrote a few more SQL queries, extracts the results, and then passes on the results to the large language model and ask it to create the rest of the report. So now let's head back to our presentation. Um let's look at the obviously it created nice reports, five such reports, but there must be a cost associated for the agent to think and execute this task. So the cost translates in terms of input tokens to the last language model. As you can see here, the cost for those tokens obviously is a function of which region are you using, which last language model are you using. Always refer to the AWS website for the latest pricing. But based on today's cost for 3.7 sonnet, it's $3 per million tokens multiplied by the input tokens gives you this much cost. The output tokens was approximately 10,000. And these are the cost per million tokens. That translates to these many costs to generate five reports. the total cost was approximately.70. So even if you were to generate 5,000 such research reports per month, your costs are just $700. Now compare that with um human costs that would be needed to generate 5,000 such reports. That would be far more than this. And also the time it takes to create 5,000 research reports would be far more than what you are seeing here. For five reports, it's less than 3 minutes. So that's the benefit of using AI agents to even analyze big data um in Amazon S3. So the key benefits that we saw here was that um anybody from analyst to CEO can create or uh research reports or explore data in a completely self-service manner and they can generate new business ideas and they didn't have to wait for long large data pipelines. they were just able to get it rapidly at very very low cost. Now we created the research reports but what about um uh understanding how did the agent reason all of this? So let's look at that. So what we have here is agent large language model and tool and how do they interact with each other. So user submit a research query. What the AI agent is doing is it's in using these two tools use AWS and file write that we specified passed it to the agent and we also specified the large language model that it should be using. So essentially when a research topic is submitted the AI agent breaks it up into smaller topics subtopics u by working with the large language model it reasons and then for all those various sub uh tasks it invokes these tools. So as an example, it has to first identify what are the objects or paret files available in S3. So it you uses the tool use AWS and invokes this API and then it gets the table schema and then it using AWS glue and then it uses Amazon Athena to create SQL queries and execute the SQL queries and gets the results and then it passes the results back to the large language model which understands those results and then it invokes the file write tool to essentially write a markdown research report in a markdown format. format. So this is how they interact and then it goes on for the next report and continues this process for the next five reports. So that in a nutshell is how the AI agent works with the large language models and the tools to get the job done. Let's pick another example of asking natural language questions. So the value for managers and even seuite is that they can ask natural language questions anytime in a self-service manner. They don't have to they don't have long wait cycles with engineering teams to build data pipelines and uh they can create research reports or responses in minutes instead of days and more importantly uh they can perform what if analysis to changing market conditions and competitive landscape. So let's look at an example of how to accomplish this. If for example on the big data that you had in Amazon S3 you want to figure out what are the top five hours for maximum fair in January and April of 2025 you submit that as a query you create a bedrock model and you're specifying to use claw 3.7 sonnet and then in this case we are using an MCP server to list all the tools that it has and then we are passing those tools to our agent here final tools we specified the model which is cloud 3.7 7 set and the system prompt and then we pass the actual query into the agent and executed it and we are also passing a Python ripple tool here. What it does is it will actually write Python code to generate nice bar charts and executes it. So essentially what it is not what it is all doing is that it's not only uh using AWS glue to understand the metadata and Amazon Athena to generate SQL queries and execute the SQL queries but it also then generates these nice bar charts completely automated. So what we have done here is that we took this raw data from park files and converted that into nice insights. So to summarize um the AI agents when you use them to analyze big data you get rapid insights in minutes in a self-service manner. You can generate new ideas um and to monetize the big data and it really makes it easy to consume the AWS analytic services just using natural language queries. I hope this was helpful. Uh thank you so much for watching the video.