Building AI applications that leverage your data in object storage | BRK124

Key Takeaways

[Music] all right hopefully folks can hear me um both here and online thank you folks for making the time uh bright and early in the morning uh to come in and uh for this talk uh and thank you to the folks online for tuning in from whichever time zone you're in so this talk is about uh bringing your data to your AI applications using object storage and um my name is vami Ken I lead product management for our blob storage service and our data Management Services um it's super great to be here and back in these sorts of in-person sessions uh it's been great to have all these interactions uh with folks uh at the conference um so what we're going to do here is dive in to details a little bit of background context setting about AI workloads as we see them and storage requirements um particularly around training and fine-tuning and rag uh I'm going to presuppose some knowledge of you know object storage systems in general in Blob storage uh how many of you folks use blob storage are at least pretty familiar with it awesome all of you almost all of you so that's great um so we won't do kind of the very basic you know L100 type intro stuff uh we'll Dive Right In and uh you know like any good build talk about half the talk is demos and actually showing you what you can do uh everything we cover in this talk here today is ready and available for you to try so if you're like hey that's interesting I can use that all of it is you know release GA material that you can go play with right away so we have lots of content I'm going to go a little bit fast on the slides to get you to the demos and you know kind of actually see what's happening so let's Dive In so when we talk about bringing data to your AI applications we're really talking about you know domain knowledge your Enterprises data uh you know your specific Industries public data and bringing all of that to AI applications now of course we're all familiar with prompt engineering but what we'll kind of zoom in on is fine-tuning and Rag and these are the other techniques you know fine-tuning is about training your base Foundation model um with you know domain specific data um and it's a more involved process uh but it is something that folks do when they're kind of training up a model you know starting with a llama model or our Microsoft's five models and training things up or even you know open AI models can be fine-tuned and more and more what we see is you know rag based apps where you have data that is you know a in a knowledge base it's prepared in a way that these llm apps can actually use it and gives you both inline in context knowledge to augment uh the llm as well as allowing you to have fresh data because fine tuning being involved is not something you're going to do every day or even every month so you know and kind of the difference is you see on the slide here it's about learning new skills with fine-tuning and rag is more like learning new facts as you go along so object storage and storage in general unstructured data storage plays a pretty important part uh in these things so let's start with the traditional training pipeline view right this is like let's say you have a model you're trying to train from scratch or you're one of these Foundation model Builders and what do you do so you start with collecting lots and lots and lots of raw data in the case of you know open AI it's maybe you know the entire Corpus of human knowledge or as much of it as they can get um and you bring that in and you have to clean that data up like that's a very common step that we all do and then you also have to prepare that data you're doing some form of tokenization you're doing feature augmentation you're doing all these things and you're doing that frankly the training data preparation what looks a lot like your traditional big data analytics pipelines you're using spark you're using data braks you're using you know Microsoft Fabric and you're preparing that data data but once you're done with that phase you have this large amount of training data that we then feed to our GPU nodes our GPU clusters run training jobs run these epochs and take checkpoints along the way so it's not only a read heavy data workload can also be a pretty right heavy workload because of that checkpointing phase and all of that checkpoint data has to end up somewhere as well sort of off the GPU cluster if you will and you use the checkpoints for Recovery use it for evaluation of how good your model is you kind of you're in that Loop for quite a bit of time and when you're ready you have your models and you have to then get them out for to your application so you have to distribute them potentially all over the world um and you're able to run then your applications and do your inferencing against those models this is a pipeline that many of our foundation model customers run today on object storage in the cloud including open AI right and we all benefit from having those base models there are some industries of course still which use and train use the same pipeline train their models we have customers for example Automotive customers are training from scratch with the data uh pretty interesting models around um autonomous driving but a lot of us have moved in this model of find chaining models and that you know the reason I went through the training you you can see it doesn't look that different right you've still got domain specific data now probably a smaller volume of it go through the same sort of cleaning prep and creating it but then you're using it along with the base model for your fine-tuning runs and same kind of checkpoints and you know looking at all of that and then you finally end up with this fine tune model that you can go deploy for applications so perhaps the scale is different and there's a little extra steps but the process for fine choing looks very similar and when you look at that and say hey what are the requirements for this pipeline what are the TR storage requirements for this pipeline you have a few things first you've got to get the data in there and that can be a challenge in itself just getting large amounts of data or even just data on your corporate Network you know safely securely into Azure we have some services like Azure storage mover that make it easy to do that for you during the data preparation phase you have to have the data available through a protocol that works well with these analytics engines you also have to then take the pr train data and be able to stream it out or download it depending on what the training jobs are doing to the GPU notes and like we mentioned get the checkpoints back those are High throughput challenges in both directions and lastly all of this process both collection of and cleaning of data and preparing this as well as the checkpoints generates a ton of data we're talking pedabytes to hundreds of pedabytes depending on the size of the model number of parameters size of your GPU clusters for the checkpoints Etc um and you really need a good way to cost efficiently retain that data because a lot of this stuff is becoming interesting for auditability and safety and being able to kind of go back and say what was the training data that I trained this model and what checkpoints did I go through in the process of training even when you get into the inference stage you know the challenges are slightly different and maybe a bit more manageable but you do have to think about hey if I'm revving these models often how do I think about model versioning how do I distribute the model to wherever my applications all the regions my applications are running in uh how do I think about about load times for these models if it's a cold start uh on on a GPU cluster that's doing the inferencing so you have other additional challenges that are slightly different as well so in Azure let's talk a little bit about what services exist and then we're going to zoom in on one Service uh for most of this talk so looking a little bit more broadly as I mentioned preparation looks a lot like an analytics pipeline so it works works well with one Lake and fabric um so if you have data that you're trying to kind of prepare for training that's a good place to go um blob storage and its data Lake functionality works really well uh with data bricks and it underlies Microsoft fabric as well um if you're looking at training historically you know AI training is basically an HPC job so there's lots of our customers who still look at having a high performance perance file system parallel file system uh like luster last year U we released a manage luster service talk a little bit about that uh so you might use that our more modern pipelines like I showed you unstructured data storage is just object storage because that's the long-term place for it to land anyway for retention and uh cost efficiency when you get into inferencing you definitely have a lot more to do especially with rag with structured data so you then start to see these additional Services um Cosmos DB has a lot of great functionality Azure SQL as well and both of them not only handle kind of storing your data for rag but also you know in some cases adding Vector database functionality into the services uh and blob storage as you'll see can be also very useful in this rag phase all right so let's talk a little bit about manage luster I'm really going to just have one slide on it and then we'll go into blob storage for the rest of the talk um as the name indicates it's a fully managed luster as a service uh it's a pass service you can choose how much throughput you need how much storage you need and we'll deploy it and do all the management for you and uh it really allows you to bring all your applications that were running on premises and this is you know as applicable for HPC as it is for AI training uh We've added some you know little good inte ation and we continue to work on that to with Azure blob in terms of bootstrapping those uh file systems from data in Blob or you know kind of making that data cold and being able to deallocate your luster file system and save money we'll continue to do more work around more Dynamic HSM integration there uh and it's also well integrated with kubernetes so if you're you know looking at more modern HPC Pipeline and you want to have containers uh instead of VMS you can go ahead and do that too like I mentioned though blob storage is really the core service that we're going to be talking about in this service and it's the canonical service when you think about large amounts of unstructured data the one thing I wanted you to take away and you know remember from what is blob storage and what's interesting or unique about it is we've spent a lot of years building you know what was the blob storage Foundation what we call here you know FNS or the flat name space it has has what you would expect an object storage system to have but about 6 years ago in 2018 we released um this hierarchical namespace capability which adds file and folder functionality akles you know recursive akles all of the sort of file system like capabilities and we use that to expose additional apis not just the rest blob API but a data Lake API that's used by the analytics engines it's very well integrated with data brakes today um and you know we have NFS so you can just go Mount uh your blob storage multi-protocol as an NFS file share uh and we even have support for legacy protocols like SFTP for folks who are ingesting data from other you know Legacy applications and systems and processing them with modern Cloud applications so this sort of multi-protocol single storage system really helps you avoid data silos and that becomes pretty interesting and useful as you'll see in some of these pipelines where you have data preparation followed by training followed by uh followed by inferencing and you'll see that we have added to augment this additional client capabilities as well so with that let's dive a little bit deeper into training and fine-tuning and talk about how and what blob storage does that really ends up working well for these kinds of Pipelines the two biggest things are scale and cost Effectiveness right we're talking about pedabytes of data and terabits of through throughput and we're able to meet that so it's blob storage isn't just great for cold data long-term retention data it's fantastic now for hot workloads for analytics HPC and AI workloads it really sings and there's a lot of investment that's gone into that over the last few years sort of under the covers blob storage has always been a costeffective storage tier we have multiple tiers from hot cool cold to offline storage with archive tier and you can manage your cost and do it all automatically we have these features called blob index tags that allow you to kind of group and tag your data and then set life cycle policies and will just take care of moving data for you and even up tiering it when you need to we talked about multiprotocol of course um but in terms of client we've done some really cool work with uh what we call blob fuse too which is really making it completely seamless for you to have access to object storage and your applications and Frameworks as if it were running on a local Mount Point um so that's what we would kind of like to go off and show you and U really the key here is whether you're running in VMS whether you're running in containers you can take your blob storage containers which are kind of akin to file systems or folders and just Mount them right and mount them and all you know get near posix like access from your Linux VMS or Linux containers the code is open source if there's something you need to tweak there you're welcome to do that and the core core thing here is we retain the you know tenets of very high throughput axis ability to work with lots of scale so let's go ahead and take a demo look at a demo here uh I have Vishnu here he's going to run through some of the demos wishnu go ahead thank you wshi uh good morning everyone let's dive straight into the demo let me switch screens all right uh so on the left you have a d96 Azu VM on which I'm logged in and on the right is a storage account and uh the container called blob fuse test and you can see these containers this container has uh 1 tbte file a couple of 50gb files and some more scripts and folders so we're going to try and mount this folder into the VM now to do that I need an empty directory so I'm going to I've created one let's see if it is empty all right that's the folder that we're going to mount let me go ahead and mount this folder now okay it's that's as as simple as that let's run that command again and there you go so it's as simple as that you just mount it you see all the files on the right immediately on the left it's just like working with any shared storage and you can actually go ahead and create a file and you'll be able to see it immediately so that's it it's that simple and if I refresh it you will be able to see the file right here oh my God I'm sorry I created an MNT I have to go ins side the mount and then run the same thing all right well there you go oops okay there you go so it's as simple as that you mount that storage you're able to work with files now this is great uh how performant is this so to check for performance we're going to try and max out the bandwidth that is available on this d96 PM which is about 35 gbits per second uh that's across both incoming and the outgoing bandwidth now before I do that uh I'm going to show I'm going to have endload uh here which is going to monitor the live Network traffic and I'm going to scale that inload in such a way that it represents 35 gbps okay so this is the live Network traffic on the d96 VM and I'm going to copy a couple of large files to see if I can max out the incoming bandwidth so I have a couple of 50gb files all right there you go so as soon as I hit enter you're able to see that blo to is able to immediately pick up the files from the mount and then immediately copy it and it is maximizing the band that is available on the VM and if you have a larger VM you will be able to maximize that as well so we have we have made a lot of improvements that we'll talk about in Blob fuse 2 that has these pre fetching and caching mechanisms that enables enables you to max out this in in coming bandwidth now this is not just the incoming bandwidth and the file just finished we just copied 100 GB of data and it's not just the incoming bandwidth we can also maximize the outgoing bandwidth so I'm going to try and write a couple of files the same folder two of them there you go so the outgoing bandwidth can also be maximized using blob fuse too so if you're copying really large files either in are out of blob out of blobs then blob fuse to does that job really well so we've done a lot of improvements uh to The Blob fuse do to enable this process and I'm going to quickly talk about these cashing options uh that's going to that's going to give you a better idea of what just happened okay so we we've always had a file caching option in Blob fuse 2 uh since the beginning like even the blob F V1 had a file caching mechanism uh and in Blob fuse 2 we introduce it with file caching mechanism so what what this does is it chooses to it it chooses you you can choose a folder where you can bring the data from the storage account you download the data to that folder and then it serves to the application so every time a file request is made that file is downloaded to the cache and then served to the application this is very useful when you have repeated reads like when you're reading AI training data sets you want to make sure that the data is available and then you read it repeatedly that's great but as the data gets larger and as you would like to as you as You' like to read the files as in when they are coming up like streaming then there's obviously uh false false shot so that's where we introduced the block cashing with streaming and that's the demo that you saw where we are able to use the in-memory cache uh effectively by downloading all the blocks of the file in parallel prefetch them and then serve to the application as a stream now you can configure the block size the number of blocks that you would like to prefetch as well as uh the number of parallel threads that you would like to run in this prefetch now this is really useful because I have a d96 VM and I can use 96 CPUs if I want and if you have a smaller VM you can always bring it down or if you're running another application you don't want to stretch the CPU so much you can always configure it so you can use all of these option and you can max out the band that is available on the VM that you're running in now this is great and when so we know that blob fuse 2 is easy demount uh you can access terabytes and pedabytes of data directly on your VM and you can max out the bandwidth that is available on your VM now how are we going to apply this to a scenario that's more recent which is fine-tuning and checkpointing scenario right so I have a small script that is going to demonstrate fine-tuning um if you see great let me just show you the script first and then we'll go through it okay um yes uh I did use stat GPT for a bit of uh coding help uh so I I'm I took a gpt2 large model uh from hugging face and then downloaded it uh Bel load that model uh I use a very random text uh a very random text to fine tune it because fine-tuning is not something that I wanted to show I wanted to show the checkpointing part so we finished the fine tuning uh we run through three box and then we checkpoint this fold we checkpoint this file directly to the Mount point that we have so the data is going to load fine tune and then checkpoint it directly to the storage account via block VI let's do that okay before I do that I'm going to go ahead and have the endload view here so you can see the live Network traffic okay so it is loading the model uh and then if you can see it loaded in just 2.66 seconds that's because it's already available I've done this before the demo and the model is already there on my buffer cache which Bluff Us 2 can use uh it can go through the fine tuning pretty quickly because it's a large VM and it immediately checkpointed and that blip that you see is what is what is US checkpointing 3 GB of data very quickly so yes if you have a larger file you will still be able to write to uh you know right to your storage account directly because there's no process involv it's just in memory cash and then directly to your storage account all right that's my demo for B fuse too this is super cool um Vishnu and you know one of the things that yeah thank you um it just and the key thing is it just works uh the super cool thing is blob fuse has been out there since 2017 we've been like thinking about this for a while and it's kind of like all of a sudden like as this stuff is lit up we're like wait we can use that stuff we went and built blob used to addressed a lot of performance issues learned a lot about these patterns of fine-tuning and data fetch data write and really spend time kind of thinking about and launching the block based caching I think you can go use this all of this is out there today just go look and GitHub for blob fuse to you can go use it on your machines so wishnu this is awesome but uh obviously we're all about big scale and uh I want to know how we you're going to show this off on a large cluster because that's the fun part right like you know it's using massive amounts of spend that we shouldn't be spending but uh let's let's talk about how you did that yeah so I know you asked us to try and maximize whatever throughput that you could show so yes we were able to maximize a single VM so we actually tried to uh you know run this a large Benchmark on one of our one of the regions that we had so we actually did an iur Benchmark uh we took up uh we took up a few AKs we we took up an AKs cluster with a bunch of PODS and we tried to maximize the bandwidth that was available at that moment on one single storage account container right so we picked a few d96 spot PMS of course we don't want to disturb customers so we we chose spot PMs and saw how much we could go we used 350 AK spots that's about 6 16,800 cores and we mounted all of these pods uh to access the AKs CSI driver that uses Blu used to and we worked with around 2 paby of data uh so what the I test does is it writes whatever data that you would like and then reads it all back as fast as possible so and then we see how much the storage system is able to handle this load right okay so so that's what we did wshi and we did manage to get some you know good I want you to notice 177,000 course um our leadership hasn't figured out what the bill for this yet is so we're hoping to have a good demo and forgiveness later so let's see what we were able to pull off okay so this is the S Snapshot from last week okay so as you can see we were able to I mean you can see the graph we were able to read about 36 37 terabytes of data in 1 minute this is a minute uh and read all back in L half this half the time like 80 terabytes in a minute uh but this this is just the overview uh but I want to actually show you the Peak Performance because we did this test a few times um and I want you to guess what speed we were able to achieve of course this is based on the uh you know the available bandwidth at that time but onei maybe why don't why don't you Takei or even S I don't know how about uh yeah hundreds of gigabits maybe a terabit per second yeah that's the right unit we should be talking about terabits here so uh the peak through put during this time was 8.1 terabits per second for just the Ingress and it was almost double that with the 13.5 tvps with and we stopped because we don't want to disturb customer environments uh and we our spot VMS keep getting dropped uh in fact we were going to run it live but you know spot VMS so we don't want to risk anything else so yes that's how much you can push with our storage accounts and with blop fuse too especially mounted on AKs CSI storage thank you that's awesome thank you Vishnu yeah look at those numbers there and this is just off the shelf like we're looking at releasing this as a sample so you can go test this yourself you're going to run into core limits faster than we will but you need that many cores to push the system and it'll keep scaling in our largest regions this wasn't even one of our largest regions uh we again as Vish said wanted be very cautious not to go above the sort of bandwidth at which we would really start affecting a customers we were running this Vishnu and a couple of our colleagues were running this overnight at like 4:00 a.m. or 5:00 a.m. to kind of stay out of the peak load on the uh in the US uh as it came online uh but again the key takeaway here that AKs cluster was using blob views nothing else it wasn't like some crazy fine-tune like really well-written code to go paralyze and download it was just containers with persistent volumes talking through blob fuse all right so what's behind this right because there's got to be something interesting happening behind the scenes um and what we've done is you know this work we've we've worked on for years now uh and it's really come into its own kind of on time and it exceeded our expectations we call those things scaled accounts and the idea here is we are spreading your reads and your rights across the entire region that your storage account is in we're taking sort of these slices of each storage cluster scale unit that we have are able to scale your account across all them so you're talking about thousands of front end know you're talking about hundreds of thousands of uh hard drives and we're able to kind of seamlessly do that and the best part is you as a developer don't have to do anything to kind of go configure this this just works we realize that we're throttling you we Autos scale you and we give you that higher throughput as you need it uh and there's no real change to pricing or anything like that you know of course you pay for transactions and all that but this isn't something where you're like paying an arm and a leg for you know high performance storage this is just your regular old blob storage regular old blob storage pricing so we'll switch gears a little bit and talk about Rag and we have a lot of content here so I'm going to go even faster um as I look at the time um I think all of you know what rag is it's about adding domain knowledge at the time of query in line with that query the pipeline for that looks different some of it looks the same you know you again have domain data knowledge base you clean it you now chunk it so you have to choose these chunk sizes because that's the context you supply to your llm that those chunks have be prepared they have be you know run through an llm these embeddings are generated vectors are generated they're stored and indexed in a vector database uh and then when your query comes in that query is run through an LM same embedding kind of vector thing and you run a similarity search that's actually what vectors are right and when you get back the matches to that similarity search you go get the chunks from wherever your chunk store was you return that you add that to the prompt and then you feed that to the and says oh here's a better answer right so that's what's actually happening underneath the covers and the requirements are different from training and fine-tuning you know of course you still want to bring data but it's usually Enterprise data more secure data that you know want to bring very new interesting different requirement is you want low latency access especially to those chunks because you are now in line in the application your user is waiting there they've typed in their prompt and you're like well is am I going to get an answer or not so you need to be faster you need to be well integrated with this Vector database capabilities uh because you don't want to be spending a bunch of time wiring all these pieces up uh you also need to support Fresh n updates these knowledge bases are only as useful as they are current so as data is generated wherever you know industry and domain you're in you want that capability and last but not least you really want great security um and access control for this data because you don't want even inside an Enterprise what the CEO gets back as an answer with all the data is not what I should get back as an answer you know as an answer with not all of the domain knowledge that I should have access to so that all needs to be handled automatically too so what we do with blob storage here the multiprotocol stuff helps for low latency access we have this fantastic service called premium blob storage that's been in GA since 2019 uh it's fully SSD back much lower latency much better for this scenario uh we have great integration with aure AI Services as you'll see in the demo and you can bring your own uh infrastructure pieces your orchestrators your vector databases for freshness we have change notifications like all the you know all file systems good file systems do but we have also have this really great change feed functionality that is guaranteed strongly consistent in order log of changes uh is unique and it is something that supports batch updates of your knowledge base so if you're like kind of adding a bunch of knowledge every day or week or whatever it's great for that without having to recrawl your entire knowledge base and lastly the security works really well because of how deeply entro ID Azure ad uh is integrated um and uh you can use arbac and you can have all your normal organization roles and all that so the concept of me as a person versus my CEO is already embedded in there and you have attribute based Access Control you can use these index tags to say hey vami has access to Project X but not project Y and that can be codified with tags inside the blob storage system so enough talking let's do demos sarup take it away Thani um hope everybody can hear me um so in this demo we are going to talk about how blob storage deeply integrates with Azure AI Services right so just to give you a sense of the setup here uh this is a familiar diagram that vami walked through but what we've done is we've now put AI search as the vector DB open AI studio is our orchestrator for retrieval and response and obviously all of this is powered by Azure openai 35 uh GPD 35 and embedding models so let's dive in I'm just going to switch to our demo screen here um all right so what you're looking at right now is the familiar Azure open AI studio uh and right from here I can dive into chat playground and before we get things started just to make sure we don't have any kind of knowledge base let's type in a rather ous and fun query what is grit in AI well gave me the English definition of grit and applied it to the context of AI pretty fun but not exactly accurate um now let's go and see how we can augment the the context with our own data source so I'm going to click under add your data add a data source and right here in this drop- down I can see Azure blob storage right as simple as that I can just click through add add in my blob storage resource which I'm going to do right here and let's say this one I'm going to pick a blob storage container uh and as you can see here I can now pick an Azure AI search resource from right here uh but not just that I can also go into AI search itself which is where I'm right now I can click on any search service and I get this pretty interesting option here import and vectorize data which is essentially combining the chunking process that vami talked about where you need to break down your documents but also vectorize them in line in one shot right so as I click on import and vectorize uh I can see the option of block storage right here and you also see one Lake as well so if you want to use one Lake you can also use one lake with what we call Integrated vectorization right so so it's very simple to set this up now what I've done is I've actually added some of this context um to my chat playground the data source has been added in in this is the same query that we just looked at let's type this query again and see if we able to get a more specific answer for our data source what is grd in Ai and there you go grit actually stands for generative representational instruction tuning uh from research work that recently happened and how did we pull this we actually pulled this from three references which I can see right here and if I click on one of these it actually shows me the parameters hopefully and I think the references are probably let me just pull this there we go so uh it's pulling it from this particular PDF file which I have right here and this is my file and this is exactly the section where this is pulling in from uh and it can actually answer some pretty detailed questions based on the chunking size that you select so there you go I was able to hook up blob storage with AI search open AI Studio and essentially perform rag this is awesome s um and uh really amazing how well all of this stuff works together and you can just kind of go get started right away again all of this is available for you to try today but s of you know of course we at Microsoft pride ourselves in providing a great platform that people can bring their own components to so what if I want to use Lang chain and what if I wanted to you know use pine con as my Vector DB can I do all of that with blob storage oh yeah absolutely Ely um in fact let's just switch to our slides here and uh let's look at we just saw the one Lake integration within AI search uh just like blob storage uh so now we're going to do a demo where we are going to replace our Vector DB with pine cone and our orchestrator with Lang chain right this is a very popular developer framework in orchestrator um the rest of the setup Remains the Same and for this time what we'll do is we'll actually dive into a bit of a code right so uh we're going to switch to our demo screen right here and now what you can see is vs code uh what I've done here is to hook up a sample application that leverages the Lang chain packages the pine cone packages and and essentially what it's doing is it is doing whatever integrated vectorization was doing behind the scenes uh with some code from line chain and pine con so let's go to this function here process PDFs from blob and here I can actually see I'm using Lang Chain's recursive character text splitter and oh by the way I can also use the Lang chain Community P PDF loader to load PDFs in and then split them into chunks that's essentially what's happening out here um of course the process of generating embeddings uses open AI text embedding models but the fun part is I can process and store these chunks which is happening right here um and then I can generate embeddings from those chunks and this is where I do the index. upsurt how many of you have used spine cone before okay perfect so this is a familiar command to you uh index. upsert is what is used to sort of upload those vectors and then when I generate my response I'm essentially generating embeddings from my query um getting matches uh from my pine cone uh index which is essentially this function here index. query and then against those matches I'm retrieving the chunks from blob storage which is happening right here um and then appending those chunks to my context and that's actually happening right right here retrieve chunks. append uh is what is chunking is basically appending all the chunks to my prompt and that is essentially what is sent to open AI um models and that is what generates the response so let's see this in action uh let's do query what is just for consistency what is grit in Ai and it got matches chunks retrieved and it actually gave me a pretty detailed answer along with the references here as well uh and that essentially is is an application that works perfectly fine with the Lang chain orchestrator and and pine cone behind the scenes um and all of this was done with premium blob storage behind the scenes and as you can see you can see all the references and the chunks um as as the answer shows up and that's one of the key takeaways it's super awesome that you can do this uh but you're still retrieving those chunks from Lobster you want that to be super performant now uh Sarah we also talked about change feed as a unique feature for blob storage for batch processing how much code is it or how hard is it to integrate change feed into this application I'm glad you asked that question vami it's super simple how many of you have used change feed let me just ask that question okay so let's let's let's maybe talk about how to sort of quickly enable change feed that's very simple um you go to the data protection Tab out here um and then you can enable blob change feed from right here you can also enable change feed from code uh but the code to essentially pass the change feed which is uh an a a consistent guaranteed and ordered log of changes and if you're looking at billions of files doing this otherwise which is to keep your knowledge base fresh is so hard because you have to keep a catalog of what changed what modified diff the catalog and then figure out what you need to process but you don't need to do that with change feed and and this essentially is the piece of code that shows that I actually get the change feed from my blob storage account and I can actually filter the change feed based on a start time and end time so let's say you're processing things at the end of the day you're a finance company there are a bunch of Trades that need to be committed you can do that very easily with change feed and then for every event in change feed you can just filter those events based on when a blob was created or updated or even deleted right and that allows you to know which indexes within your vector database do you need to update and So based on those updated blobs you regenerate the chunks again using premium blob storage and then index. upsert the vectors for those chunks into pine cone right so it's it's as simple as that vami so it's what maybe less than 10 lines of code maybe 20 lines of code but that's that's pretty much it that's awesome s and uh you know it's an interesting thing that as an aside the reason it's buried under data protection is we built this for doing backups of a blob storage so we we ourselves needed a technology to kind of say how do we know what changed without having all this stuff worked and then we're like wait this is actually kind of useful for other people too for batch processing not just in AI on not just in this rag scenario lots of places where you need to go kind of figure out what changed in your uh object storage system and go use it absolutely and I think one of the things which I would love to talk about is the fact that we were using premium storage behind the scenes for these rag applications and premium storage as vami talked about it as well is SSD backed and so what that does is compared to standard storage which is typically what most people use for transaction heavy applications premium delivers 3x faster rag performance at scale uh and not just that it actually delivers 65% savings on transactions now as you can imagine uh for billions of files when you break them down into chunks you're writing those chunks reading those chunks it is a very chatty application because you're interacting with storage that often and when you do that the number of transaction blows up right and so you want to save that and that's what premium delivers so not just that the fact that you get speed with premium but you also get those savings on transactions so um that's pretty much pretty much it I'm that's awesome sarup thank you so much and premium blob storage again very proven quantity has been GA since 2019 people use it a lot for interactive scenarios which of course this is kind of an interactive scenario when you're quering things in line works wonderfully well very widely available in most of the regions you're probably writing your applications in um so we're at the end of our talk almost on time we'll be here for Q&A afterwards as long as you want us to be uh or as long as they can soon as they kick us out we'll have to leave uh but uh you know I want to kind of wrap up here with what we talked about in this talk we talked about how blob storage is ideal for AI training and fine-tuning that scale the cost Effectiveness all of those things that you've known to grow you know grown to love are here it's really well integrated with the you know analytics functionality that you need to prepare your data uh and blob fuse 2 is a fantastic addition that allows you to use it for fine-tuning with your GPU clusters uh as if you were working with u you know normal mounted storage um it's also great for building rag based llm apps U we've been talking about premium blob you should go try it out if you haven't um it's integrated really well with Azure AI services so if you're building these applications you can get started in you know less than an hour um it's interoperable with all these other Vector databases and orchestrators and um it's really easy to have freshness either you know in a streaming fashion with change notifications or badge fashion with change feed uh and the core thing here which we didn't get a chance to show given this length of the talk is how well that entra ID integration works with both our back and this attribute based access control so if you're looking at you know securing access and doing sort of scoped querying and scoped access to your broader knowledge base for your application definitely look into this it it makes things a lot simpler to have that integrated versus having yet another system about that's checking access and whatnot so that's our talk uh thank you so much uh for coming here and spending the time with us and we're happy to answer any questions you have e