Language Models: Developing Apps - Challenges & Solutions // Barak Turovsky /MLOps Podcast #169 clip

Key Takeaways

I'm bractrovski I'm executive in Residence at scale Venture partner a leading Enterprise say BC BC firm focused on AI and I I'm not a big connoisseur of coffee I just prepare a cappuccino using a machine so there is something that you you kind of talked about there on one hand everyone needs to remember that Infamous blog post that is you are not Google but on the other hand I would love to hear what you as a product owner what you think about when you are looking at these new applications that are coming out and as you mentioned there's a big difference between making a Twitter demo like an auto GPT that blows up and gets a ton of stars but then when people actually use it and try and use it in production it's like dude this is not working and that can go for any kind of app right now or any use of these llms and how we can mitigate some of these big questions and trade-offs that we have one being costs that's huge right and then the other one that you mentioned is latency and so yep if you are trying to do this at scale and really have a battle hardened Bulletproof llm Ops pipeline what are you looking at and how are you thinking of that and especially also just to add another question onto the infinitely long question how do you see other companies doing this well first of all I think it goes back to my framework is that people should be realistic what can be achieved and start these skills that I would call them easier than harder because cases because if you try to shoot for the start of the Fate I'll replace Google Search right that's very difficult llms might take 10 years together because there is always you know the Pareto Principle that in productizing something you sometimes spend 80 to 90 percent of the work to to optimize for five percent of the use cases so for example reducing hallucinations you might try taking literally 10 years to get from 90 accuracy from 90 to 95 and that might not be enough because on Saturday 19th 99 accuracy on the flip side if you need to create if you start this creative use cases I don't know or I can get movie script you might be totally fine with 80 accuracy right and 20 Hallucination is less of an issue so I think first of all you need to be realistic what use cases you should focus on right that's the first thing and then there's still ton of topics that you mentioned one of them back to the use cases is that like latencies and cost believe it uh guess what even on those you should focus on easier use cases because many use cases latency is less of an issue it is a huge issue in search because people expect instant answers so you need very high accuracy means probably means larger models bigger costs and you need very low latency mini and very very high freshness meaning in search you need Super Fresh results otherwise there's a stone of such queries that are influenced by uh by fresh results and guess what fresh results have much more abilities too either been hallucinated or being influenced by Bots or whatever if tomorrow I don't know Trump was injected or Trump announcable running for president you can have ton of boards that will completely pollute your results right and it's not even hallucinations it's for real something something not not fully uh not fully correct if you go to use cases of I don't know please create a draft email for me a you probably can wait maybe maybe even an hour you can wait so latency is not as uh rigid uh as I said accuracy results are not that critical uh Etc so the first and foremost I think classical rule focus on use skills that are achievable and then even on those youth because there's still a work because you need to make sure also even the point of you know open-ended nlu based input it sounds very cool or let me allow people to say whatever they want but what I discovered in my many years for example worked on speech that our language is extremely rich and it's awesome but it also extremely open-ended and ambiguous and I always give people this example of you know if you were a witness of a crime scene you will you call to a police station they like describe the describe the suspect like well I I cannot describe I cannot draw like don't worry let me bring you a police sketch artist he will help you draw this person and then you suddenly discover he's asking you these questions that is you know 15 types of eyebrows and 15 types of ears and 15 types of eyes and 40 types of noses and then it becomes very complex so in many cases when you give people a lot of choice to express themselves it's awesome but it also complicates the product solution in many cases you actually need to provide people easy to use either toggles or you know UI choices you know like shorter versus longer story visual versus a textual story um casual versus professional so a lot of the things can also be simplified there are still ton of things to be figured out there but make big advice to people and I think we can also talk about the indices where I believe that will be disrupted focus on use cases that early lamps are are good fit today versus use case that llms could be a good fit tomorrow because there's potentially a very long road to make it happen [Music] foreign [Music]