Using AI for RegEx on Telemetry Pipelines | Amazon Web Services

Hello there. I'm Nolan Chen, partner solutions architect for AWS. And I'm Ozan al-NU, founder at Edge Delta. Ozon, earlier in my career, when I was a Linux programmer, I remember I often had to write complicated and tedious reax expressions to make sure we collected and filtered the right logs, traces, and metrics from our Linux systems. Can you tell us how AI today can now make that job easier? Absolutely. Let's take a standard log file or even a log message within a file. Reax and logs are as old as time. So if we think about a regular log file, uh they're going to have lots of different messages. And each of those messages is going to have different fields or attributes within that log. And if we can identify the various different attributes, one thing that you typically did as you probably have a lot of experience with is you wrote regular expressions to either filter, parse or transform that data. So let's take an IP address for example. Maybe you might want to cut out that IP address whether you're masking or hashing that data because you don't need it for a downstream uh system. So that is a very typical thing that you might do and you're going to have to write a regular expression to match that uh piece of data. The other thing you might want to do is pre-agregate. So you might want to look at what is a specific error message. What are the various different metadata or attributes associated with that? And you might want to pull out that error message as a different field as well. The challenge ends up being is that you have potentially billions of logged messages coming through. Those could be in hundreds or thousands of different formats. And as a human to have to sit there and manually write all these regular expressions has been quite difficult. You have your team that moves on, new systems are onboarded, new sources are onboarded, new destinations have various different requirements. And so one of the things that pipelines can actually do is give you a tremendous amount of value in automating this process. So one thing that Edge Delta has done is it takes small samples of your data and will actually feed it into AI. What AI can do with those samples is recommend certain things like masks or filters or hashing certain information and will actually apply that within the pipeline as a recommendation for you as a user to accept. So again in one of those instances going back to the first example it can be an IP address that the system has identified and said this might potentially be PII data. Do you want to filter it out? You can apply that filter right here and one thing that happens is it automatically applies and all the data going out that information is masked at that point. This is one way where if you're going to send this data to S3, you could be sending it into your SIM. You could be sending it into, you know, an alerting system or Slack or Pedager Duty or Service Now. There could be a lot of various different systems you're sending this data to. to be able to have these AI insights that work in stream in real time to give you recommendations for how you want to be transforming or filtering or optimizing your data sets. That's one thing that we've seen to be extremely valuable for the engineers like yourself who might have been writing reax for decades but might not want to do that anymore. Thanks Ozan. The hard part wasn't actually writing it. It was trying to analyze all that data to figure out what to write in the first place. So if I understand correctly with AI pip AI powered pipelines in the middle, you can now automatically get those recommendations. I can look at those recommendations. I still have to manually implement it myself, but now I've been saved how who knows how many hours having to pour through thousands and thousands of lines of telemetry data to figure out the right reax expressions. Absolutely. And I think today you have to manually implement it. Maybe tomorrow we automate that part as well. But I think that's the key for the user such as yourself. If you're the person who's the system administrator, if you're the person who is in charge and you it's your responsibility to ensure that all this data gets from source to destination in the right format, then of course you want to be able to use a tool like AI that can recommend certain things and then you get to be the master of your data. You get to choose any source, any destination and specifically in any format. Thank you, Ozan. I appreciate you appreciate you telling us today about the power of AI and how it can help you make better reax expressions for your telemetry pipelines. Thank you.