Deploying LLMs

Key Takeaways

welcome to data skeptic machine intelligence our podcast series exploring contemporary topics in artificial general intelligence and large language models in case you're not the developer type let me tell you getting access to these large language models and using them programmatically in your own code it's shockingly easy in the same way that you can go to chat GPT or Bard or Bing and talk to these language models you can just as well in most languages write about one or two lines of code and get the same response but that way you can do whatever you want with it invent the next great software solution launch the next great social media platform and although there's some really exceptional exceptions of people using these tools very well in Solutions and products right now it's fair to say it's early days so in the next couple episodes we're going to cover more about that deployment Story how are business is adopting these Technologies what does it take to do that to engineer that although I described it as quite easy there's absolutely a few challenges we'll get into that more right after this I'm Aaron R and I'm aanad Global CTO hey this is Priyanka here and I am uh based out of aanad Southeast Asia leading the AI iot offerings and I'm also the Microsoft mvp4 Ai and can you tell me a little bit about the company what do you guys do yeah so so aanad is a services company we are a joint venture between Accenture and Microsoft so we serve clients and help them across all Industries around strategy technology enablement and then really thinking about the customer employee experiences and how that wraps around the technology of the business challenges and problems we're trying to solve uh any technology in particular that's exciting right now you're working on there's this little thing called uh Ai and generative AI you know there's some new things that we are doing in the immersive space as well and then there's a just ton of things happening from the data front and in terms of generative AI uh I think people kind of get it we've all tried chat GPT but could you share a few more details maybe some ones you're excited about where generative AI can be used no actually you know we have had a very successful go live with one of the largest oil and gas companies with southeast Asia and that's the poster boy for geni implementation in production and U using the security of azure open AI it has been incredibly you know rewarding for them to implement generative AI uh on their own data platforms documents so they have immense documents which are you know just lying across disparate data sources unstructured semi structured images videos audio clips and U the kind of uh productivity gain with the help of generative AI being able to you know find insights nuanced Communications from the documents was uh another level altoe so the productivity boost was more than around 40% across their Enterprises within a span of 6 to 8 months previous to gen you know the same experience was like not too much great for end users or the kind of productivity gains was not too great and cut to gen and the immense know phenomenal transformation which they saw even for us without the the new State ofth art models coming in right so that time when we went live which was in late August gp4 was not yet available so even with the Lesser so-called powerful models even with those they could see a phenomenal exponential productivity and uh uh you know ux gains could you expand on the core of where that 40% % Improvement comes from like what actually was saved right so a lot of you know and and this is coming from the you know the exact verbatim Court of the CIO and the CDO of that organization she wanted to democratize the data right and give the data power to the hand to the hands of the end users so a lot of the end users they they spent at I'm searching for very mundane queries right like uh where can I find this part how can I fix this failure here because a lot of that information resides uh in with the field Engineers it's not well documented it is just based on the experience of those field Engineers so how to free this data from you know the confines of the users knowledge or somewhere lying in Legacy documents and actually uh liberalize the data to empower the end users so the routine searches itself uh you know where people spend a lot of time over and over again getting wrong results or or uh searching the same thing that itself those gains were so phenomenal and then again another productivity area was where you try to learn from past experiences so was this done somewhere earlier where you know in instead of Reinventing something can I reuse can I can I base my uh you know new implementation on my past learnings so that is another area where it kind of had tremendous games and again Incident Management so searching for incident tickets and resz utions so that was another area so the the searching time the kind of experience they got from uh you know their past learnings so that is the areas where they found a lot of gains and processing time also document processing with the help of jna just to Baseline a few statistics previous to J let's say 500 documents used to take 3 months and with Genna 500 documents take three hours wow well I couldn't read 500 documents in three hours uh does that get QA or what how does it work now in this new world no it's like before the documents are exposed out for question answering you need to process the documents and the information from the documents is uh you know stored into different knowledge stores so you could have the documents transform into knowledge graphs or some of the information from documents actually extracted out as images for thumbnails or some of the unstructured information going into cognitive search indexes so with the help of uh generative AI it is so much easier with the help of just natural language prompts to sort of teach the model like or given this text you know this is the kind of onology I want to extract out of it and this is the kind of uh knowledge graphs I want to uh build on top of it it's just so much simpler yeah there's um another client example that we've got who is in the retail space and for their Frontline workers so those retail store associates we in a very similar Type of Way looked across almost 21,000 different documents that they had and different types of data set so think Salesforce data other corporate Enterprise data and created a experience for those retail store associates to be able to quickly get access to information that before they had to hop through two or three different systems to be able to be to do and now we've got sort of this really kind of easy user interface to be able to extract that information and knowledge for them does an idea like that does someone come to you and say we think uh this could be done or do they come to you saying what can we do with these new technologies we see both sides of that we have seen from some of the early adopters in different Industries different models that we are working on or maybe use cases and then how we are actually sharing those because I'm a big believer that in this space that we're in right now I mean let's just reflect and step back for a second like we are only sort of one year into open Ai and the release of Chad GPT so as much as there's been in the technology space specifically a lot of conversation around what do we do what's the value how do we see this I am a believer that we should be really open about well what's working and what's not working and as a part of that then we're able to go to a bank and say Hey you know in Europe we were working with this particular bank and they were thinking about these different use cases and their parts of the business and share that same kind of theme or general areas with say in a different part of the world to be able to go well this is how they're doing it now obviously each organization is unique and they have their own challenges around their own data structures to be able to make some of these things happen but it truly is democratizing this in some way where we're doing this ying and yang of sharing between hey they're thinking of a problem we've seen some of that problem and how do we kind of help each other go solve that the space is evolving pretty quickly what challenges does that present in terms of deciding to roll out now or maybe wait a few months and see what's out there let's use Microsoft co-pilot as a great example here just look at the last sort of eight months of the features that Microsoft has rolled out from the initial announcement of what was M365 co-pilot and all the functionality that sits in there to be able to provide these services to each of us as a user of let's just call it you know the different M365 Suite of Office Products we fundamentally and I fundamentally believe that co-pilot is a significant change that any of us that sort of working today has seen probably since the 1980s with the introduction of IBM in the spreadsheet like it is fundamentally going to change the way that we all work enabling us with a tool and copilot is much more powerful than a tool but enabling us sort of with that to be able to then just turn it on and Kyla give it to you and then you're able to use it that's all great great but how do we make sure that there's something enduring that's there and that there's value that's a part of it and so part of our approach is actually thinking about the people side of this first and how do we then go figure out so like in our own roll out and in the work that we do with clients and try to get them to think a little bit like us in this way which is where you're kind of putting people before the AI and how do we make sure that we are thinking through the communication and the actual change management that has to come as a part of this do we even have the right culture in place to for everyone feels comfortable to be able to utilize these tools in various different ways and so I think there's a ton of interesting things underneath that's enabling this but we've got to also make sure that we are truly thinking about us as individuals and how we're taking ourselves along this journey that is truly kind of a change in how we are going to do our work were you given an AI problem to solve with too little data or too little time I know I have web AI here's you they're a team of AI and ml researchers and practitioners who've created Navigator an IDE designed to streamline the mlops process to get your project across the Finish Line in hours and days instead of weeks and months with state-of-the-art architectures like deep detection and attention steering you can build object detectors and classify ERS as well as expert conversational agents you own your models and data forever and can even train and run locally to ensure privacy and security when you're done tweaking integrate your Creations effortlessly into production ready Solutions full code like I like or drag and drop deploy to any environment at the edge or the cloud from ad hoc proof of Concepts to full scale Enterprise deployments Navigator has you covered get early access at web aai /d skeptic So within the last four or five months we have seen models getting more and more ever powerful so we started with gpt3 then there was GPT 3.5 turbo now it's GPT 44 turbo gp4 vision and we know that this is going to keep on getting you know more and more powerful right so probably what I was trying to do for image searches a couple of months ago that is now offloaded to GPT before Vision so which is why a lot of customers also want to wait and watch regarding the evolutions most of them wec are in a stage where they want to experiment so but but stay at the POC or MVP level and while productional liing they want to keep an eye out on the developments and that also goes not only for models but also for the hardware the compute is the most essential for all these models to to perform the way they do today so the kind of Hardware changes also so uh the offerings around different comput instances different on Prem open source uh sorry not on Prem open source models open source small language models large language models so there's a lot of Paradigm shifts happening out there and which is why a lot of customers are not in a hurry to take the next you know production step so they want to experiment they want to get a feel of things on the ground but then they also want to wait for those huge announcements to be made before deciding their next step to go into production we're at a time of kind of a fundamental architecture shift and if I just look at large organizations and the IT infrastructure that exists and the applications that are there the way we've been thinking and designing this co-pilots bring a change in that because so I I'll use sort of an Avan not example of our roll out of um Microsoft co-pilot four years ago our it group built for our own employees is a chatbot that then you know basic kind of questions like how do I reset my password what's my how much vacation do I have left those types of things okay it was not the greatest user experience it had its challenges but it was the beginning of kind of what we saw of kind of where this would go turn that to what we're doing right now with Microsoft co-pilot where Microsoft has the ability to make connectors to different data sets so for our instance of Microsoft co-pilot we have a connector that we turned on to service now because all of our it knowledge based articles are based in service now now I can just go to the team's co-pilot and ask a question of hey I forgot my password and it surfaces the knowledge based article from service now and we didn't code hardly anything to make that happen whereas look what we did just a couple of years ago while we're talking architecture the underpinning of all this is data traditionally we have thought about any product or service that's being built we have our and again I'm coming from larger Enterprise sort of perspective on this we have our kind of core Enterprise data our CRM our Erp our other kind of system data that we have and then all these things that we do around our collaboration and the data of what Microsoft calls kind of the Microsoft graph around our documents we we really kind of treat those things separately we do figure out the integration that has to happen around those but really to realize the things from a generative AI where co-pilot is going and actually this evolution of work that was talking about you really need to start joining this data up because you've got this context of all of the applications and the ontology of the things that we do within our own business you then have all the things if I just use myself as an example all the documents and the meetings that I am in and the emails I am sending and then if I want to actually complete some tasks the marrying of those two is almost the holy grail and so how do we actually begin to bring those together and co-pilot studi is the beginning of kind of seeing how this Vision sort of is there but I go we're we're super early sort of in that journey and I think it's going to you know we're with these early adopters and so how does how does somebody a year from now who's doing all their planning think about do I have the right data foundation in place do I have the data where it needs to be at the speed that it needs to be and how am I actually thinking about the the my my people and my graph data combined with kind of this Enterprise data and these new products and services that we want to build like it's I think it's a super exciting time but also a challenging time because we're in not just change of work but we're in change of habit of how we've designed things before when you're talking to people about strategy no one's going to say our strategy is to stay away from AI everyone wants to get into AI yeah but what are the actual practical opportunities here today in 2023 back in January through I'd say March April time frame in a lot of way of what prianka was saying we have a lot of clients that were doing all right what's the use case that I want to try and I'm going to do a lot of experimentation and kind of these proof of Concepts to try to figure it out not really with a strategy in mind just a we're going to we want access we kind of want to see and learn as a part of that and I would say overall we're probably at like 40% of clients today are still in this experimentation kind of Po kind of Realm of the things they're doing whereas back in February that was probably like 90% over the summer so I would say you know kind of the May to July August time frame there's actually a lot of clients that took a step back and go okay we are on pick your favorite buzzword I will just say our transformation Journey because there actually a lot of clients that are on their way to get to the Cloud some are not even kind of in the cloud yet and then some are much further so they all need to think about where they're at on that journey I don't think of AI as a brand new transformation it's a way to go all right in our strategy of what we're doing where do we apply this new set of functionality that is at a scale we hadn't kind of thought about before and optimize our business in some way or look at a business process that maybe we can now reimagine in a way we hadn't been able to do that before there's been a lot of work on like well what is our strategy kind of even at a board and sort of a CEO level that then has been assigned to all right there's some line of business leader who has a clear use case that they've done a business case around and has value and they're actually designing and building that for scale that work is kind of still happening and is going to happen over the next you know several months to years but that if I kind of put a percentage of again 40% kind of in experimentation they're sort of probably right now about another 40% that are in this like we're kind of Designing a something that we're building for scale that utilizes generative AI to be able to do that and and they've got a strategy around it um that is not just a technology strategy but it's a people and a change strategy as a part of that a much smaller percentage is so for the remaining kind of 10% or a little bit less than that how do you begin to use gen as a differentiator that's where like the super early adopters are at the moment of going like we see something we're sitting on a lot of data Maybe how we monetize that data in a way we hadn't kind of thought about before do we maybe want to invest and build our own lot our own large language model or something around that and fine-tune it a whole lot I think that part we're going to see grow over the next 12 to 18 months some of them are waiting for stories and I think we're so early in this journey that we haven't shared enough of what others are doing and as we do that we're going to start to go oh well that's interesting and then the majority of the organizations who are not kind of these early adopters will go all right this is how we're going to start to flow it into our business process in some way well the idea of a custom co-pilot is really appealing not only do I get the benefit of the Native co-pilot but somehow it knows the intricacies and nuances and dirty secrets of my organization what does it take to get something like that built if I were to put the effort into perspective uh you know earlier with the with the earlier versions of generate AI somewhere around you know four to six months or six to 9 months to get into production but then depending on the kind of maturity your organization is in so probably they are already you know onboarded the cloud journey and they have a decent data platform in which case their Runway is smaller right but then organizations who are on Legacy platforms yet to um sort of you know grasp the power of a centralized data platform so for them the runway might might be a slightly you know longer because of the kind of processing power because of the kind of intelligence and logical comprehension gen brings in what what transformation could have taken as x amount of time is definitely reduced by half with the help of gen so organizations who are on Legacy platforms are not too modernized still if they want to look at significantly upgrading their platform and then building AI applications or custom co-pilots on top of that uh with the help of you know the Gen processing part they can look to significantly reduce the time for actually building these kind of applications and custom co-pilots again you know you have variety of ways of bringing them into reality so identify the data sources identify the use cases I think the use case and the goal for which you want to harvest that custom copile that is the most important thing so as long as the organization is clear on what is the end result the kind of productivity boost or the challenges they want to mitigate with the help of the custom copilot it becomes easier for SI like us to go and then actually help them you know uh reach that end goal and build that custom copilot the way I like to think about where we're going with co-pilots is think about when the iPhone was first released it was not immediate and I'm going to have my time frame wrong in this but it sort of feels like maybe a year or two after the iPhone that's really when Apple kind of created and bolstered the App Store I look at co-pilots and what Microsoft is doing specifically as a whole new Dev ecosystem with that Dev ecosystem we are going to be in a spot where today we maybe call them plugins or custom co-pilots I think a nice way to kind of think about it is we're just going to have these inapp experiences of a co-pilot and things that we are doing the way I think it's going to come to fruition in the short term is in a way that there is an app store and we saw this with the announcement about a week a week and a half ago from open AI is that they're going to have for lack of better term I don't know what they called it but there's going to be a Marketplace that's a public type of marketplace and I don't have all the details and there maybe some other things that they announce as a part of that but I I believe that there will be Enterprise Marketplace for an organization so I'm just to make it simple let's just use Avan not as an example we have a Marketplace of custom co-pilots or plugins that exist and then those will be by Persona so let's say we have someone in our finance organization who is using Excel they're going to use Excel out of the box from Microsoft and the co-pilot that kind of comes with that that's only going to get them some percentage of the way through the tasks of the things that they've got to complete we then may build or there'll be another software company that provides this that then we're able to kind of host in this Enterprise App Store to be able to go oh I want to be able to link this type of data to you know another organization and I need it to query these different aspects or whatever it may be I'm going to click and grab that skill or Plugin to help me complete and do that work and I think there's going to be the ability for us to almost have a curated way to kind of look at how how these different again going back to the change in the way we're working there's going to be other aspects that are going to be there to help me in that but not just in the pure kind of out of the box because it's not going to have that Enterprise kind of context or understanding we're going to have to figure out how we marry those two things together and then build the experience around it well you'd mentioned the possibility of organizations training their own large language models my initial reaction is that this is may be inaccessible incredibly expensive uh need a lot of compute and it's more than just terraform apply you got to figure out how to do that you need the data and that maybe it's just the the domain of a the proud few that get to do it am I being too pessimistic I agree with you I don't think you're being pessimistic I think it goes back to that differentiation is it worth it and what do we get and is that is it worth it I mean to use something super simple do we build borrow or partner maybe there's a case to be made in some instances where it does make sense to go build something and and maybe that changes over time with what comes with some of the Innovations from either the open source side or the cost of some of these things being able to come down the other side of it is I think there are so many possib abilities where how do we leverage what exists from different providers and maybe that is azure open AI um maybe it's something else but that you really then are getting into significant finetuning to kind of make this work in the way that you want to and it's a different level of investment sort of as a as a part of it what's right for your business and where you're at and I think the majority makes sense with today's Tech to use what's out there versus build your own but I will tell you we do have a couple clients that are interested in doing this and you know we work through that strategic conversation with them around well is it build by or partner to kind of get you where where you need to be because of the speed we're talking about this how do you make sure you're not doing this and then there's something that comes out in 3 months that you go well now well I don't know if it's worth it anymore yeah true before before we even talk of custom llms of Enterprises just let's take a step back uh 3 months ago right so organizations were still you know not clear whether to uh use the open AI versions of GPT models or to have fine-tuning of those models and so on and so forth right and then um once the customers got a bit educated they came to a conclusion that oh fine tuning all the time is not needed I don't need to custom train my own models I can use the retrival augmented generation pattern and um you know do most of my tasks 9 95% of my you know chat with your data or kind of find insights with your data tasks can be done by by the RG pattern so again yeah I mean you you need to do that tradeoff way your Investments versus the ROI you are expecting but yeah I mean that's a demand from clearly certain sections governments for example defense sector health and public care where they would want to have or medical sector for that example for that matter where they would want to have a very you know custom llm train for their purpose for their uh you know usage we we so we might see an increasing Trend but then again to what extent and uh to uh you know how much would be the demand for it or how much would customers go to uh those kind of end results remains to be seen and uh as we've said The Field's moving really fast what are you both most excited about in the near and long term for the for the near term right I'm pretty excited about the kind of possibilities this is going to open up for our clients even for us as an Enterprise right so all the uh different co-pilot Studio announcement or it's Microsoft fabric going GA and you know Maring this data platform which is tailor made for the era of AI so these announcements they excite me tremendously and just looking into the kind of possibilities of what it means for as as an Enterprise as as an SI or for our clients that excites me tremendously and again multim modality so generation of images and audios and videos what scares me for the long term are two things one is how this is going to be uh used right so for example you know your creation of videos would that uh sort of encourage the creation of deep fakes or you know your cyber crimes secondly is because of this whole implosion of uh your huge Computing parts and data centers how sustainable this is going to be because we talk of uh you know all these billions and trillions of uh parameters hyper parameters ever inre increasing um you know Computing uh capabilities and Computing Parts but then what does it mean to keep a data center running like that constantly 24/7 and with you know new data centers popping up for for sort of uh Bridging the ever growing demand between um different SI and clients so how sustainable that is going to be and how that would affect as in the long term so it's a a two-pronged you know worry but then it's a double its w i mean might come to a point where gpts themselves are self- sustaining could be so it is interesting it remains to be seen how these challenges we we battle them and we mitigate them yeah I I also think with responsible AI every organization while they need to think about the policies and guidelines that they have in place actually having principles is something that's really important and one of our core principles from an organization perspective and that we are very clear on is around being overly transparent when it comes to AI so you know there was talk about sales co-pilot um we are an adopter of sales co-pilot but when we started going through that adoption cycle we realized this is actually generating now it's not Auto sending but it is generating an email that then a salesperson is looking at and modifying slightly and then sending our privacy policy and us being really clear that AI actually autogenerated that email we needed to make a policy change and we actually needed to do something that you know many organizations kind of have a footer that goes on every email that says privacy policy etc etc avad has never had that what we've decided to do is for those people that are in sales co-pilot at the moment we now have amended a footer so that when people are getting that email they know and that's kind of coming back through we have this principle and we're trying to live and breathe these values around the things that we believe from responsible Ai and that's why I think for every organization it's really important to not you you can run fast but you need to also be thinking about and having all of these kind of guidelines and governance in place around it and it's a really great place to be able to actually take these values as an organization that sometimes maybe just sort of sit on a website or are espoused and sort of certain meetings throughout the year you could actually live and breathe those things through AI in a very unique way that actually empowers kind of the rest of the organization as well we also need to talk about the kind of governance that needs to be imposed the kind of uh you know ethical responsible AI practices that need to also be out there in the Forefront so it's not only about implementations but also about the responsible usage of the AI and having a robust governance in place it is your data government governance it is your AI models governance so all all of that also needs to be emphasized and talked about so what goes wrong if it's not yeah so uh there's one you know very popular thing uh which we in now our legal also do a lot and which is called as indemnifying your generative AI results so we we at the end of all our uh gen AI implementations or you know the results which we gave out we always have a disclaimer that this is AI generated and please exercise due caution before you act on the suggestions or recommendations of AI as more and more the AI models generative AI models get powerful it becomes imper that you have these kind of Frameworks for the for the end users to be able to trust the outcomes of the AI models and to have that boundary where you know we recognize that it is still people first so which is why Microsoft has very beautifully given the moniker co-pilot so which is like they are enabling you enabling you uh to do better enabling you to you know be more productive so that thin line between uh the fact that we still control the models that is where the governance things comes in and of course security around the data data leakage confidentiality because imagine where you know Finance sector start using llms which is very um uh you know discrete data about customers pii data same for healthcare medical sector so in that case this kind of governance confidentiality becomes even more significant and which is why having this kind of you know security ethical use becomes even all the more all the more important imagine a case where you have a medical bot now how do I trust the advice given by that bot uh given my symptoms hey you might have pneumonia there has to be some way for me to trust that so it's not that every other company will be able to have their own model creation and put put out there for the uh you know use of the end users there has to be certain criteria by which you are productionizing your models are they safe to use are they safe to interact with the end users how do end users trust them so there has to be this entire framework where your model is assessed for fairness for ethical usage for responsible usage before you make it available so I think with the growing demand these these practices need to be more stringent and they need to be enforced and is there anywhere listeners can follow you both online um yeah so you can follow me on my LinkedIn handle which is uh CPU uh and also on my Twitter handle which is @ fuzzy mind1 it's it's it's a slightly funny moniker yeah so on Twitter I'm I go by the name of at mindman I also have my own blog on medium uh medium.com which is at theate AI geek and you can just follow me on LinkedIn and you can just find me with Aaron R sounds great we'll have links to all the above in the show notes for people to follow up thank you both so much for taking the time to come on and share your work no thank you very much enjoyed the conversation thank you [Music] Kyle