Distributed AI Architecture: Core Infrastructure Principles for Enterprises

These days, AI is everywhere now. Whether they're looking to optimize internal processes or improve customer experience, maybe even gain that competitive edge, enterprises are looking for ways to bring AI into their existing infrastructure. Now, AI manifests in a number of different ways. It could simply be a GPT style agent that they're putting on the front end of an existing web application. It could also be an internal agent that's there to augment existing human processes. Or maybe it's a protein modeling uh AI that's there to help with drug synthesis. AI exists everywhere in this spectrum. However, one thing is consistent. Enterprises are looking to drive real value with their adoption of AI. Today, we're going to start to explore the concept of distributed AI. Now, this might be a relatively new term, but it talks to this idea of moving AI closer to end users and to where data is generated, ultimately improving user experience and faster decision-m all whilst upholding security and sovereignty along the way. Now, all things with AI begin with data. And for a lot of my customers, they have relatively fragmented data landscapes and largely that comes from a long history of operating. Now data itself when it comes to AI is the fuel. It's the thing AI needs to be intelligent for that organization. Now let's take the example of a payments processor. They might have data that exists in collocation as part of their core processing systems. They might have a bunch of microservices out in AWS where they've made use of Cassandra or S3. They might even have systems sitting up in third party SAS providers. Maybe they've got their customer database of records with someone like a Salesforce. No matter where these data localities reside, for AI to be successful, organizations are going to need to bring them all together. Essentially, a way to converge data from all of these different systems. And now, this idea of data sprawl is not new. AI simply exposed data sprawl, although it didn't create it. And so organizations are trying to trying to solve for these data management challenges. But now with AI, those agendas are being accelerated. This all refers to data that's under the control of that organization. However, a lot of my customers also interface with third parties, partners in their ecosystem, partners who provide their own data to enrich this organization's existing data sets. For our payments processor, this might be fraud scores that a third party is aggregating. No matter what the approach is, organizations will bring this data together. And once that data has been brought together, an organization can then start to evaluate how they add AI. Now, in the context of our narrative, AI is the intelligence layer. It's that thing that's going to make all of my different data sources useful. And as an organization, I really [clears throat] have four different options when it comes to building this intelligence layer. My first option is to simply partner with a model as a service provider. You could think of this as an open AI or as an anthropic. My second option would be to simply take an open-source model off the shelf and run that in my existing infrastructure or on some new infrastructure. Now neither of these are necessarily aware of the data in my organization although some approaches exist to add that context to these existing models. The third option is to take that same open-source model but to tune it on my data. So essentially refining an existing model. In this way the model becomes a bit more aware of my context as an organization. And my fourth option is then to train an exist an entire model myself. So essentially starting from the ground up with all of my data. Now for a lot of the customers I work with and those customers that operate in heavily regulated industries, the first two options might be nice because they're simpler to start with. However, it does create some concern around where my data is being sent and also how these models were built in the first place. And so what we see is customers gravitating towards the last two options here which is to tune on their own data or simply build their own model. Of course this depends on the use case and what the organization is trying to achieve but for more proprietary AI systems this is where our customers generally head. Now in both of these cases we're going to start to see the requirement for more intensive infrastructure and generally that comes in the form of GPUs. Now customers have two options when it comes to deploying this infrastructure. The first like any cloud service is to take it as a service. There are plenty of providers out there today who can deliver the latest generation of GPUs to customers on demand. And the second option is to build essentially procuring this hardware yourself and putting that into a high performance data center that has the space, power and cooling to run that infrastructure. Now, whichever of the four approaches a customer takes, the output is going to be the same. some model that understands their business and is able to fulfill the use case with which they started this AI journey. And this is where we really get into the distributed nature of this story. Once we have that model, the promise of distributed AI is to move it closer to where decisions need to be made. decisions that impact customer experience or decisions that impact how quickly we can draw insights from new data coming into our platform. And this is where you need edge locations that are close to both of those sources of data. And so this idea of inference or inferring from that model that we've created is really where we see the business value. So this is where organizations start to reap the rewards of all of this stuff they've done before from the data modeling into the intelligence layers and now to the business value. And when we talk about distributed, this means this model might live in multiple locations, the same model, but wherever it's needed to create that business value. And so we could imagine that maybe we have regional models, one that lives in AMIA, maybe one that lives in APAC. But these choices are then based on where data is created. and where users reside and that's this idea of generation. So where new data enters our system. Now this could be a user logging in through our web application and that web application is then in close proximity to this AI system. In the context of our payments provider, it could be a point of sale terminal somewhere out in a retail outlet that's looking for fraud detection in real time. Again, this model needs to be close to that infrastructure for that response to happen in real time or near real time. And so this is the premise of distributed AI that we can push a model that understands our business closer to the things that are going to be used in that model that drive business value either from a customer experience perspective or from a datadriven outcome perspective. Now this is not a oneanddone process. This is all new data that's being generated that's ultimately going to flow back into our existing data infrastructure. And so this AI deployment is actually a continuous feedback loop. As new data enters our system, it's going to get stored. That data then becomes fuel for the next iteration of our intelligence. We build new intelligence in the form of a model. That model is then distributed back close to the consumers and creators of data. And this cycle perpetuates. Now there's three things that must be true for this system to actually operate and drive business value. And the first is careful consideration to data placement. Data has both gravity and inertia and that means it can be hard to move especially large volumes that are generally needed to drive AI. And so organizations really need to give uh careful forethought to where and how data is placed. The second is that the network is an imperative. The idea that data needs to flow in near real time across different boundaries while still remaining sovereign and compliant is quite a difficult challenge to solve. And so the network itself needs to be high performance, resilient and also secure such that customers can move their most trusted data across these platforms. And the final one is that ecosystem is important. ecosystem in the context of partners who are going to help enrich our platform with more data and ecosystem in the context of providers who can help us run these workloads with the latest generation of infrastructure. Now, I know that's a relatively short and quick introduction to distributed AI, and maybe it's generated more questions than answers, but that's what we're here for. So, if any of this has prompted questions that you might have about your own initiatives when it comes to AI, or you simply want to learn more, feel free to reach out to us here at Equinex and we'd be more than happy to help you.