Trends compilation Agent built using Responses API, Computer Use Agent iin Azure OpenAI

Key Takeaways

Hello all. This is a demonstration of a trends compiler agent. Um the scenario is this. A user would want to uh look at what is trending or latest on a particular uh topic on attire and provides this as an input and then this agent takes that information and then automates a lot of browser actions. So in this case I have used Pinterest as the source of all this information. So through the combination of the computer use agent model of Azure OpenAI and using playrite um libraries on my local computer and I'll be showcasing how we can automate those steps in which the user input is translated into a search query and then u we extract information from Pinterest and then create a single document out of it. So u without further ado let me run the application. This application uses the responses API of Azure OpenAI and uh which has the integrated tooling for computer use agent and um uh MCP server. I'll talk about the MC server part as we as we go along further in this demo. So now very shortly we'll see the browser opening up and then taking us to the landing page of Pinterest. The computer use agent will instruct playright to click on the search tab and then whatever I had submitted as a user input has been translated into uh search text in the search bar. Just hit enter on that page and then it has clicked on the first image that came up in the search results. You'll notice that uh there are actions happening here in the scenes as the program is running. It clicks on each item or image takes you to the detail page as a seen now and then it uses the reasoning capabilities of GPD4 to describe the trend aspect of what is seen on the picture in all its detail. You can see that uh it is navigating to each detail page and then once the information is obtained from the screenshot, it navigates back to the search results page and then moves on to the next image and so on. The back you notice here that the reasoning model has given a detailed description of uh what is unique about the image and what kind of trend or fashion statement is depicted there. In the configuration to the application, I have I have uh set the number of images to consider as five. Even though Pinterest might return more results, I'm just taking the top five in the results and then compiling the uh information from it. So all the file images have been processed. So now the agent would consolidate all the information and create a markdown document which contains the trend information captured from across all those images. So now it's creating the markdown report. Okay. So there it is. It's come back to me asking me for the next step in the whole process. But let me quickly uh scroll up here and then walk you through. Yeah, as you can see this is it has created a consolidated information from across all those images. So let's say I want to store this information catalog it somewhere so that I could reuse it. So now I'm going to ask for certain information for example where do I need to catalog it? So um so at this point the agent determines that uh it needs to uh in call the um MCP server which is um configured as a tool action. to this uh agent. So there you go. You see that it has come back by making the MCB call invocation and then it has given me a list of all those containers that are found there. Uh so now I see that uh this particular uh container by name trends compilation seems to be the right one where I want to catalog this report. Let me go ahead and use that. So uh you notice that it has performed this operation. It has uh created a markdown document with this name and then it has stored this in the blob storage location. So if I were to navigate to that now I should be able to see it there. So let me refresh this. Okay. So you see that the information has indeed been stored uh in the storage location. But just a bit about the MCP server itself. What I missed showing you earlier was it has already integrated and running. So I could from VS Code itself uh show you the um the MCP server that is already running. So when I navigate to the agent mode and get a cop pilot and when I click on this you see that uh the uh the MCP server is up and running. So I can click on the tools action and then you'll notice that uh here is the MCP server that I connected to and these are the various actions that uh the server has. So in our case we used the list containers to list all the uh containers in the storage account and subsequently we called the upload blob action to upload the report that we created. So this concludes the demo.