Azure AI Content Understanding: How to Get Started

Key Takeaways

Hello everyone and welcome to this first video on an introduction to Azure AI content understanding. This is the first of a multi-part series where you're going to learn more about Azure AI content understanding. In the other videos, you can see more about how you can use content understanding to rag over a video data set or build a post call analytics workflow using Azure AI content understanding. We're excited to have you here. My job today is try to help you build your first project to build content understanding into your applications or workflows. So let's get started. Content understanding is a set of capabilities for extracting content or structured insights from documents, images, audio, video or content of any modality. You start by defining an analyzer that allows you to extract either the content that you can use for your rag workflows or the task defined schema that is a more deterministic result for your agentic apps or your automation workflows. Let's get started with content understanding in the AI foundry and walk through the process and I'll give you a little bit more detail as we go along. So here I am in the new Azure AI foundry. The first thing you'll notice is in order to create a content understanding project, I'm going to scroll down and select on content understanding here. And once I have content understanding selected, I'm going to create a new project. Content understanding uses the new Azure AI services resource. And in order to do this, you're either going to create a new Azure AI services resource or use an existing one. I've got an existing resource, so I'm just going to go ahead and use that. The resource is then connected to the AI services hub as well as a storage account as in your project creation workflow. So I'm going to go ahead and select a storage account that I already have. And now I can go ahead and create this project. This step typically takes a minute or two. So I'm going to move over and I have another tab created here where I've got a project created. This is like any other good cooking show. So here I'm going to go ahead and add my first file. In this case, I'm going to select a PDF document. Right? So, the first thing you'll notice is by providing a sample content, content understanding will now give us back a set of templates that we can work with. In this case, I'm going to get the document analysis template. I'm going to go ahead and choose that. Um, once I have my template selected, I can then go use uh the field extraction capabilities. Just remember that the template is nothing more than a configuration of some predefined ways of how you can use the content understanding capabilities. You can use this or you can choose to start from scratch and build your own. So I'm going to go ahead and first define my first field. Uh you'll notice that in this particular case since I'm working with a document, you'll see the document on the right. For those of you familiar with it, it's a 10K document or essentially a annual report. Uh, one of the fields that I might be interested in extracting from this particular document is the company name, right? And you can see I can give it a field name, give it a little bit of a description. What am I trying to extract? Give it a strongly typed field type. In this case, it needs to be a string, but it could be a variety of other types as well. And finally, I choose an extraction method. The extraction method could be extract, generate, or classify. And when you choose generate or classify, you're essentially telling the model that you you're expecting a value that is either synthesized from the from the content but not necessarily extracted from the content. Now that now that I have this defined, I can go ahead and hit save. Um, this is essentially the process that you go through to create the the fields that you're trying to extract. Notice that you're not labeling anything. You're just defining a schema. Uh so what I'm going to do now is I'm actually going to go ahead and and skip forward to a existing project that I've already built a lot of these fields into. Right? So you can see that now I have a a lot more fields here and these fields are things that I would expect to extract from a 10K report. Some of these things are pretty complex and you'll see that content understanding is now going to be able to extract these values for me from these documents. The next thing that I do once I have my schema defined is I go ahead and say analyze this particular document. Right? So I can test analyze it. You can see as part of that that that first analysis, it's extracting all of the fields that we were expecting. So company name is extracted. You'll also notice that it's producing a confidence score with each of these extracted values. These confidence scores are essentially the way that you can use to identify if there's something that you need to evaluate as a human or you can straight through process these documents. And this is a great way for you to save cost, right? So as as you increase the rate of your straight through processing, you're minimizing your cost and and the system is behaving in a in a in an optimal fashion. So now that you have your your workflow defined, this is the first thing that you've you've done is you've gone ahead and created the schema, you've analyzed your documents, you can try to continue to do this, add more documents, continue to test. Once you're happy with the with the results, the next thing you're going to do is actually create a analyzer. And like I mentioned earlier, an analyzer is nothing more than a a bundling of all of the information of the work that we've done so far. So we created the schema, we define the document type or the content type and all these things are essentially configurations within the analyzer. But what the analyzer does is it produces a a REST API endpoint that you can start using. So now I'm just going to call this my annual report two. And so this is a quick operation. And it takes about a second or two for the analyzer to to to get created. But once this is done, I now have a REST API that I can I can invoke and and test test out my um my documents with. So I can go ahead and hit test. And now I can upload a file and I can actually use this analyzer as a REST API directly in my projects, my agents or my workflows. These are all ways that you can configure a analyzer to to sort of drive insights into your applications when you're using content understanding. In order to show you how this might look, what I did was I go I went ahead and created a aentic app. Right? In this case, it's an agent uh that uses this exact uh endpoint that we've built and it's using that as a tool within this agentic workflow to say whenever it's asked to work on a 10K document, it's going to use this particular tool to extract the fields that that it needs in order to perform the actions that it needs to. So in this case, I'm going to submit a upload a document and I'm just going to ask it to extract the fields needed from the document. Right, I'm going to go ahead and hit send. Okay, the results just came back and you can see that the agent is now able to work with this particular document using the tool that we just created, extract the fields that we needed and produce the output. This output can now be used in a downstream agentic process. For example, you could do things like, you know, compare it against industry averages, produce a more meaningful report. All these things are available for you to use because now you have content understanding that deals with the challenging task of working with all these content elements and producing an output that means something to your industry or your business. That was a quick overview of what content understanding is. What I wanted to leave you with a few additional links in terms of how you could continue to to learn more. Obviously other videos in the series, but try out the documentation or the samples. And of course, if you have any questions or feedback, please do contact us. That email address is as well on your screen. Thank you so much for your time and attention. We're excited to see what you're going to build with Azure AI content understanding.