Cross-Cloud Management Made Easy

Key Takeaways

yeah so Envoy AI Gateway allows you to be able to run cross cloud like hybrid cloud and and that's one of the big features and one of the reasons that the problem arose as well is because all these different Cloud providers have different ways of accessing um their system but yeah I mean kerve itself is you know similar to something like stag maker Google vertex if you and incloud you can definitely run K serve as well if you're running if you want to manage your own you know inference Services I mean it's definitely helpful if you don't want to use one of these uh Cloud providers if you want to maybe if you're worried about cost savings and you want to manage something yourself a lot of people can use it on cloud as well but I mean also there are these products for inference running inference services that are also very similar have the same goal So ba so the envoy AI Gateway is and just stick with me here because I know I'm a little slow on it but no no you're good first time I I'm really digging into it um and I really like these AI gateways it's not the first time that I've heard of it but I do find that it is a problem folks have especially when you get rate limited so easily if you're using external Services you want to have some kind of a fallback plan and so it's almost like Envoy AI gayway is an abstraction out and you can throw whatever endpoint you you need underneath that so whether you're using a sagemaker endpoint or a Vertex endpoint or a k serve endpoint you can link those all up to the envoy AI Gateway and it will figure out where the request needs to go depending on what it is yeah yeah exactly yeah it has a unified API where you you can uh easily specify you know what you need to do and it'll Auto R for you which is great it's all about just making it easier to run and easier to manage and yeah you start to see patterns like I said everything's kind of starting to be built on top of kubernetes and built on top of these other tools all with the goal to just make it easier to run and not have to worry about the infrastructure so much I mean there are a lot of also really cool or really interesting problems brought on by these uh large language models in gen systems as well the model sizes are so large the downloading the model takes a long time as well so one interesting problem that kerv is starting to work on is the model cache being able to cash models and not have to download them every time a pod starts up for every pod so that's something like little things also that will be super super helpful like GP working on GPU utilization as well of course because gpus uh right now are are resource limited so a lot of these issues around running and getting enterprise Enterprise AI up and running being useful and being optimized is something that we're definitely working on [Music] baby b