Fireside chat #9: Language Processing: From Prototype to Production

Key Takeaways

hi everyone it is Hugo bound Anderson here I am so excited to uh welcome you to today's fireside chat about natural language processing from prototype to production within as uh montani uh we'll get started in a few minutes uh in the meantime if you're able to introduce yourself in the YouTube chat uh and let us know where you're calling in from um uh what your interest is in machine learning and natural language processing uh and moving from prototype to production what type of work you do uh that would be super cool um and we're we're triply quadruply as excited as as usual today uh because this is our first fireside chat uh that we're doing in the European time zone so for everyone who's missed out previously so excited to have you here today and we'll get started in a couple of minutes foreign hi everyone it's Hugo ban Anderson here very excited to have you here today so great to already have uh just shy of 80 people online uh excited for our fireside chat with Inez montani about natural language processing uh moving from prototype to production and all the different things about NLP um it's really cool to see people dialing in uh from from Berlin from Paris from Morocco from Dublin from India from from Nigeria people who work in uh NLP um and also work in other other types of machine learning and data science um Ivan from Switzerland working on utilizing nlu to work with uh intellectual property documents fascinating um we have people working in Industry we have ravenesh from Dusseldorf John Germany who's a master's student so we have people working people studying people learning um people working in information retrieval this is all incredibly exciting hey we've got someone from Melbourne Australia as well great great to have you here and we have Jennifer Sampson from London an NLP practitioner working in energy as well so um we're going to take a couple more minutes for everyone to get in hi Igor from the Netherlands great to have you here um and we'll get started in uh two minutes all right all right everyone it's time to get started I'm I'm too excited to to wait any any more minutes um so without further Ado um it is what do we turn our cameras on and have a chat yeah hi hey how are you good how are you yeah good good morning everyone in um Good Evening in Australia and good evening in Australia and good day or good day as we say um from the whole Spectrum in between I actually had a friend once who in an Australian friend who was actually born in Israel but he um we're in Europe together and he tried to turn g'day into different European uh versions of it so we actually we're in Berlin and he tried to say guitar to German people and he just got and he got the most ridiculous Stairs Of course and he tried bonjour in Paris and but yes in Spain the Spanish were most receptive to padillas yeah this stuff only works in Australian English yeah exactly Australians as well yes yeah exactly um so it is so great to have you here in us today um such a pleasure and I just want to welcome everyone who's joined we've already got 100 over 120 people have joined already so very humbled and excited that you're all interested in in such things um so we're gonna chat for an hour uh or so and then after we're actually going to have um an async ask me anything so if there are questions that arise please put them in the chat everyone uh but if we don't get to all of them feel free to come to our slack here which I'll paste in the chat um and it is and I will both be answering any questions you have in there's a channel there called ama ama guests um and oh we've got we've got someone a bunch of people from Australia as well um and we have a number of people saying Kudos such lovely things it's kudos to the team developing Spacey and Prodigy Rock Solid and great documentation which um well thank you and I I actually I was I was going to get to this but as you know I I work on the metaflow team and at out of bounds um and the people who started metaphor my my colleagues actually would were deeply inspired by by Spacey in the early days of metaphor in particular the documentation and the human-centric design has just been an inspiration and they they were sorry yeah yeah definitely things that were important to us that we put a lot of work into so yeah it's always great to see when these things resonate absolutely um and so just a little bit um it's we're hosting this fireside chat and at out of bounds we work on kind of similar kind of philosophy to to you um explosion and on Prodigy and Spacey we we work on infrastructure and productivity tools for data scientists to kind of allow data scientists to focus building on models and doing data science while having easy access to all the infrastructural layers um and if you're interested in checking out anything we do I actually I work in devrel and build a lot of educational material um so I'm actually going to put in the chat sorry someone's just commented when I said bonjour that Hugo your French is perfect that is that is brilliant um so I'm putting in the chat some resources for data scientists that we have where we really think about the full stack of machine learning um and we also have um a lot of tutorials including an NLP tutorial so I thought to drop that in as well if anyone is is interested um and the obligatory um hit subscribe and share with friends if you think it will interest them um wherever that I mean it's down there down there somewhere um but without further Ado I'd love to I'd love to jump in and I'll I'm going to introduce you um but then it would be great to kind of hear about your trajectory and what you're working on now and you can correct me in any of my introduction but um it is Montana I'm sure you'll do a great job um well you helped me with the bio I think as well so is a software developer working on AI and NLP Technologies um she's the co-founder and CEO of explosion where they make Spacey one of the leading open source libraries for NLP and Python and Prodigy a modern annotation tool for creating training data for machine learning models is that did I do pretty well that's that's how I would summarize uh what I'm doing exactly yeah and on top of that we do a lot of work in open source a lot of related libraries we're currently working on um an extension product for Prodigy which runs partly in the cloud and addresses the need of scaling up annotation projects and data collection projects collaborating um so that's coming very soon I know we've been saying this for a long time but like it's really true now um fantastic that's very exciting and um yeah we're always interested in helping people solve real world NLP problems in the wild um and also yeah using um the latest things that come up in research and then translating them into something that can be useful um or seeing what what's interesting what's promising and once it's more mature how can we use that in creative ways to make people's lives easier amazing I'm so excited to jump into all of that in particular kind of how to bridge the gap or how you think about Bridging the Gap between NLP and research and what happens in production and in Industry I know there are a lot of online resources for people to do NLP in in notebooks and and and that type of stuff but then what is actually how does how do you move from prototype to production and how those things are interrelated um but before all of that maybe you could set the stage a bit by um telling us what what's happening and what you do at explosion and a bit about Spacey and Prodigy I mean I I presume a lot of our audience knows knows about space and Prodigy but for those who who may not it'd be great to get the elevator pitch yeah sure so um well this is not going to be the elevator pitch but um I will just talk about Spacey and Furniture right um yeah Spacey is um an open source library in Python it was um initially started in 2015 which is quite a while ago and the inspiration was my co-founder Matt he did a PhD in the field before it was called Ai and he started noticing that more and more companies were interested in using his research code and um Wanted to license it and he was like well no this is like research called research code is supposed to um print a number and exit this is not what you should use in production but it basically showed like hey finally these Technologies are starting to work there's really a gap here for a library that's focused specifically on production use cases that's fast efficient has a good developer experience and that's kind of how Spacey was born and I joined the project shortly after for a long time it was a very small team developing the library um and yeah the focus has always been to provide practical components and also take an opinionated approach um provide one model or like one implementation that is the best and has the best trade-offs between speed accuracy customizability while also making it easy for power users to extend the library with anything custom and that can range from like implementing a super new machine learning model implementation um to doing something more quote unquote and old-fashioned like hey just adding some rules and some rule-based extraction because they are areas where hey this just works much better you don't need machine learning for everything but you want to combine it all in one API that you can work with so that's that was the philosophy of Spacey and um then yeah very very early on we realized machine learning it's not just old it's not just algorithms it's always cold Plus data and once you're serious about doing it you always want to have your own data and do something custom that's also where the value is like most value is delivered in stuff that's quite specific to a use case yeah and so we actually we needed an annotation tool internally um so we built one that we would like to use that's also a developer tool and that's how Prodigy was born and um that became our first commercial product and it is a developer library but it also comes with a UI for annotating um different types of data and from text to images now to even audio and something fully custom and the idea is really to make the whole process efficient so you can iterate faster really focus on hey collecting the most important information putting a model in the loop because data development it's like code development it's not static you need to keep doing it ideally even the machine learning developer needs to keep doing this we have the vision um very early on like hey imagine you have a great idea for something new you want to build imagine if you could just go nowadays you don't need so much data anymore you can collect a few hundred examples run an experiment and see if your idea works that's awesome and um for that you need a tool that works efficiently and fits into your workflow um and yeah that's that's kind of where Prodigy fits in and for those who don't know can you tell us a bit about what annotation is and how it may differ from um everyone's aware of what labeling is but how it may differ from um yeah I mean I think I think these terms are often used interchangeably but basically I mean the the rough idea the fundamental idea as well um if you're doing machine learning or supervised machine learning one idea one one way of tell you what I'll pretty much always want to tell the computer what to do that's kind of one of the great things about programming and Building Things um you need a way of telling the computer what to do and one way of that is while using a programming language another way in machine learning is to use data so you basically show um the model and the training algorithm what you wanted to Output ideally and then you train your model and um at the end of it um ideally it should have generalized enough that it can do the exact same on unseen data so that could be hey you want to apply labels to a text um you want to extract certain spans of text like person names from a text um those are all options you want to have relationships of a text or cluster Concepts in a text together like the mention of a name and pronouns that refer to that name um there's basically there's kind of no shortage of what you can do um but the important uh question is what should you do and often that's not clear just like programming it's very rare that like the first idea you have for how to write something is the right uh way and you never end up refactoring your code that's usually not true and often the same goes for you NLP pipeline there are lots of different ways to break it down and you have to find the right one yeah so this was Prodigy then when did explosion come come into the picture uh so explosion started in 2016 so kind of uh in between Spacey and Prodigy right um so we um yeah we realized like when I started working on Spacey um we realized like hey they're actually a lot of cool opportunities here for company and um we decided to bootstrap the company and stay independent um for a long time also to show like hey there's a lot of there is a lot of interest in the field there's no need uh to be running at a loss initially um we want to see what the market is like we took on some Consulting projects early on um to fund uh the development of prodigy or so to get some connection with real world problems and um yeah then we developed Prodigy um and that was the commercial product that we were able to fund the company on and then when we got serious about finishing up um or like when we you know we're at a point where things were going well and we're like now all we need is finishing up that extension for product for prodigy or kind of the next iteration um we um that's the first time we took investment and we sold five percent of the company initially to be able to hire a larger team and get the product out and over the finish line and that's kind of where we're at now great and how many people are you now at exposure um so now we're just under 30. okay at the moment so it's still I I guess you know relatively small but I think it's also for software um you know we don't necessarily believe that bigger teams did better software um I think even within like big tech companies um a lot of the software is actually built by smaller teams um need the rights team um and the right ideas that's more important then agree more and I think um edges in the graph as you try to scale a company need to be you need to be very Mindful and I've always I always ask whether 100 people can build more than 10 people and I think if you especially if you grow very quickly I think it's very unlikely that you yeah one person can only have so many direct reports like as soon as yeah yeah get more people in you get more managers in and then yeah you have to you end up delegating things further down the line and I think that just yeah it also didn't really fit to the philosophy we had also being like two Founders were very Hands-On in the beginning and we're both programmers we're both technical we're both absolutely developers so yeah yeah and I don't want to get too much into bureaucracy and these types of things but it is having worked for several startups now it is a fascinating Point um in a setup trajectory when middle management emerges and how how mindful you need to be of that emergence and who you put in those yeah those middle layers it can be a make or break time in a lot of ways um but enough about the bureaucracy of growing and hyperscaling startup life um this this is a kind of a relatively naive question I think in a lot of ways but I think it can be um fruitful for the rest of our conversation could you give us a rundown of just what NLP natural language processing is and and why it's why it's important and why it seems to have been adopted all over the place yeah sure I mean it's actually it's not even such a naive or bad question because I also feel like people often use the term to mean slightly different things so um I think I would say in general it's natural language processing is processing text that comes in in the form of natural language natural language most of you would have um guessed um so you know using a computer to analyze text and um also in ways that goes often goes beyond uh just searching for keywords um I think some people I think nowadays usually what's also grouped in to that field is um natural language generation and natural language understanding um you know understanding is more like hey we take we get this natural language in and we want to have some structured output the generation that's also what I'm sure everyone is familiar with hey you have like a model and that outputs text um or even something like translation I think that's usually seen under the same umbrella but what we focus on is much more on the level of natural language understanding and information extraction I think that's because that's also one area um that's incredibly important um across all kinds of business use cases and where even currently a lot of value can be delivered because I think one situation we have is hey we've been as humans we've been amassing text over like a very long time and as um you know we become more digital digitalized we keep amassing more and more and more texts the internet keeps growing um we have a lot of text and we want to analyze it in ways that we couldn't necessarily really do manually either because we have way too much for a human to even read or because yeah we want to extract things and connections that also are not viable for humans um to extract by just reading text and keeping track of everything um that's happening and what everything means so um and I think that's also why it's um so important um and yeah why people have found it useful we're at a point where a lot of these things actually work like really work not just like theoretically work and um are actually useful even just to automate some processes that previously had to do had to be done manually or in combination with humans because humans and machines fundamentally make different mistakes so if you're only yeah humans can be inattentive humans um can be lazy in humans um yeah humans don't have a great memory and then you have machines that make mistakes that sometimes make us go like wet like how how do you make that prediction but um when it comes to consistency um and um memory um yeah um we can't beat a machine so both of that together basically leads to better results and I think that's that's also why NLP has been pretty successful yeah great and what and I know the answer may be a lot of Industries now but what industries do you find at NLP being most relevant to and how what what type of things do they do with with NLP yeah I mean it's really as you said it's really most Industries like when people are not using our tools so that's not using NLP and I think we've also seen that when you did the quick intro of uh people in the chat we already had like five different Industries come up even just in the very beginning um but of course they're like they're more classic use cases of b2c applications startups e-commerce that maybe more people think of where hey you have product descriptions or you have texts and emails coming in from users from customers that you need to categorize analyze um that's one area we have um legal um legal has a lot of text um that's also very obvious there's contracts that's all it's also um also a field where um you know it really mistakes um can be very costly and can lead to a lot of problems so um being able to work more efficiently with large and large volumes of text um that's often also follows I would say if often a structured approach but also a very formulaic like it's a language in itself um that's also where NLP can be very successful um even just pointing out hey here's something that's weird maybe I want to check that again or resolve references to other legal Concepts um there's like there's a lot there medical um again there there's a lot of work there's a lot of studies there's a lot of material um a lot of very specific con Concepts that can be extracted you can talk about um you know research of new drugs um effects patient notes from doctors everything's texts often also yeah if you go into the sort of healthcare area a lot of it is just written in all like in all kinds of unstructured ways notes the doctor typed about a patient um follows no consistent structure and somehow you want to standardize that um digitalize that there's a lot um Finance um again it's similar use cases to legal but also there's stuff like or financial um modeling and um Financial work following the news following what's happening in the world is very important you want to extract stuff or company reports what do you know companies uh publish often there's a lot of actually interesting stuff is often hidden somewhere in fine print um and not nicely formatted in a table um often on purpose so you have like this natural natural language you want to resolve there following what's happening in the world analyzing news seeing what's going on um trying to um understand you know new trends if you're you know from federal reserve banks setting interest rates they need to know what's going on that's like to um people uh trying to find predict Supply chains um hey there's a strike in this specific area in the world and that's like 10 levels removed from something else I'm buying yeah how can this disrupt like these are these are for example these are use cases of that sounds boring but actually are incredibly valuable similarly traditional manufacturing you have a manufacturing planned for the past decades Engineers have been writing up Word documents of what's been going wrong when there was an incident and you have all of that and now you want to have put that in a database sounds a bit lame but actually huge huge value incredibly important and valuable and impactful as you say so I'm really interested in getting into now what what NLP looks like in in production as opposed to NLP in research and then as as we said before there are a lot of resources online for um learning about how to do NLP in notebooks and that type of stuff but how do these things differ from when when you're building NLP models in production in industry um yeah I mean I don't want to necessarily say notebooks are bad I mean there's a lot of not at all there's a lot of you know sitting on notebooks ultimately it's just a different Computing environment but of course I think when most people say hey what what people mean by notebooks is oh you just kind of copy paste some stuff together or hack something together to have a proof of concept and um of course that's yeah and then often the challenge is hey how do you take that and put it into production yeah and my intention wasn't to disparage you know I I think to prototype um for example um and I know people who use notebooks in in production as well I'm not one of one of those but um yeah you might know out of bounds came out of Netflix or the software came and came out of Netflix yeah that we work on and they use a lot of notebooks in production so yeah in production okay yeah yeah no I think um so basically I mean if they just look at like a lot of a lot of what we're working with in Industry comes out of research and that's like that's um super important so a lot of the ideas were first developed in research and um one aim of research is to kind of build a common of knowledge and um solve uh hard hard problems and kind of um you know come up with new ways um to solve um you know very specific tasks and um uh find um yeah find kind of way find gen have you know General Solutions work on the algorithms and um kind of on the other side you have like actually taking the taking that work and taking the comments of knowledge and using that in a real world uh setting and um often um you know there you actually have one of five solutions that are very very specific to a given problem like you have a business need that you want to solve end-to-end and you want to um yeah you want to do this and you want to find the solution uh that works best um and uh you know often you also have very different considerations for um speed accuracy um you know you or also the trade-offs between the two um and even just the the question of like what should you even do in the first place um I think often um you know if you just if people come into the field and follow both like research and Industry it can be a bit um misleading or can be a bit confusing to see latest models uh come out or like oh some accuracy numbers that are reported in papers and then trying to decide hey should I replicate that on this paper why does this not work for my thing and it's like well these are fundamentally different objectives and it doesn't mean research that should focus more on solving business problems like that's business problems are um can often be quite simple you can also you can make you can make your problem easier like you need to you want to find the easiest best solution um to your problem um and decide how um to break it down into components that you can easily solve that you can reliably develop that you can maintain um and yeah that that run and serve serve the purpose that you're interested in great and I'm I'm also interested in there there are some people asking stuff like this in in the chat um around how you think about the types of skills needed to move from working in notebooks to deploying models the the software stack that's basically interoperates with how to think about integrating your models with larger software Business Systems and and all of these types of considerations um the skills you need yeah I do think having good software development skills is often even more important than having like learning a specific library or specific Library API I think it's it's kind of like you know look yeah you it's it's almost a bit like yeah programming languages more generally if you have like if you're good at like one programming language and you try to sort of get into another one um you can often pick up the concepts uh much more easily where it's like oh if you're kind of limited to just knowing how to do a thing with a certain library that can help um you solve problems quickly but once you want to get a bit deeper um you know you can easily feel stuck so stuck so I think the kind of being able to write decent code is a good skill to have and it's also um something you could pick up and um something you know that means um yeah you have like more prospects in your role and can go beyond just um building the prototypes because otherwise you know you can just if you're building a prototype and that prototype can't actually be used at all and someone else then has to go and rewrite it um entirely to make it um at all useful that's maybe something that can work in a really large team but that's not really sustainable um in a small organization so I think that's um that's definitely helpful General programming and also um if you're working um with text I do think a bit of uh Linguistics and really understanding language can go a long way um like yes people um because no matter even if you're working with the you know large language models or the you know something that um you know it inherently doesn't look like you know you're analyzing language or you're writing rules about language understanding how language works conceptually and having like a basic understanding of linguistic Concepts can help because it also helps you design your solution better like you don't need to be you know do a PhD in linguistics and you don't need to um go super deep but they are fundamentally we do like if we're looking at natural language to grammar and the constructions in the language and the syntax that we use is a big part of how we express what we want to get across and are a part of how we convey meaning in the text so um if you're thinking about hey how do I can how can I extract certain Concepts from a text in a consistent way it can help to understand oh okay these are sort of these are units syntactic units in the text these are consistent these other things are not so if you have your model and it like outputs arbitrary phrases that can be kind of anything from you know proper nouns like a name to half sentences and just anything arbitrary that like this is very hard to compute with like yes you can maybe train something like that but at the end of it you get something out that's like does it actually help you like um how do you how do you go from there you can have a model yeah abstract everything that's interesting to you but then usually the next that's usually not the last step usually then you want to put it in a database you want to relate it to something else and so thinking about um you know how language breaks down I think is often helpful and it's often I think it's often the type of thing that some people might feel like they're missing like there is sort of or sometimes we see that when talking to people there's often this Vibe of like there is this secret source that like I feel like oh I don't I don't have for NLP that like I could take you to the next level and I think it's not actually that deep and it's often not like oh the the super in-depth details about the model that often doesn't even matter um that significantly it's it's the first step is about breaking down the problem into something you can solve and into things you can extract and how like how you go from a larger business problem to um actually an NLP pipeline that extracts something um and if that is bad then How can any like yes you can do some hyper parameter tuning and like throw some larger and larger embeddings at it and maybe you get some more accuracy out but that's kind of that's kind of not what matters here like um you know if if you they like your data needs to be good your approach needs to be good your idea needs to be right exactly understanding how your data is generated and on top of that I mean where my mind also goes is that we're kind of talking around having more domain expertise as as well right like understanding the specifics of the language you're you're dealing with um and your ability to understand it and understand what features are important if you need to do any feature engineering which you likely do at some point and that will be a lot more have a lot greater lift for you in whatever models you build than some hyper parameter tweaks yeah and just we've got a great comment that that I like and I want to kind of drill down into this slightly that um software development is underrated and machine learning is overrated relatively speaking um and what I what I take from that is a lot of the conversations we have in the space are around the powers of machine learning but um having good software development best practices is incredibly important particularly as we're trying to tell more and more of um the deployment story these days we do have a question related to this I want to get into what applied NLP thinking really is um in in a minute but we do have an interesting question around the problems technical details of the problems that one make might face uh while deploying such as debuggability versioning uh throughput optimized scoring I'm wondering in your mind what are the the big things you see when moving from prototype to production um in terms of technical challenges technical challenges um yeah I mean um like most mostly I think what we mostly focused on is is kind of is a bit starts a bit earlier before that so it's like okay we've you know we're definitely using a lot of kind of ml op stuff but that's also kind of not the area um we're in I think there's definitely yes there's obviously um ideas of like okay how do we um you know how do we yeah surf the thing make it stable how do we um you know improve latency um they're actually the trade-offs here yes people always want like the most accurate solution but if the most accurate solution is too slow that's um yeah that's a problem and then it's about finding the right trade-off um but I think also the kind of continuous um development and the kind of iterative improvement because it's one thing yeah a lot of people just a lot of talk is about like okay you have you take your model you put it into production into your fancy cluster or however you do that and have like all your nice mlops tooling around um and then it runs but then how do you go from there to improve it because most likely your data will change even the world will change like there are very few industries that are completely immune from yeah the world around us and the language changing like references change even if you even if everything you're working with is about your company your comp stuff in your company changes your products change your suppliers change like there's there's stuff that's just different yeah and um you want to have this continuous look of actually making sure um you can catch wrong predictions and evaluate them see where the problems are analyze them generate more data improve the model and then also make sure you're actually not making the model voice um this way and I think a lot of that is um there's a lot of tooling you can use around that but I think some of this again also comes down to just a more fundamental reasoning about let's say evaluation uh methods and just data like um you know if you if your evaluation data is [ __ ] you it's really easy to get 99 accuracy that does not mean your model is great so um I think that's also often something that's not talked about a lot so you do want um yeah a representative sample and representative examples that you can test your model on so that you can be really sure that changes you deploy actually make it better and not worse absolutely and there was something you you said in there that I think really plays into how you think about um applied NLP thinking you can have a very a very accurate model that's super slow and may not work in a business context right so you've got a wonderful blog post that that'll share um called applied NLP thinking except if I recall correctly um and part of it is thinking the Linguistics is actually part of it and this isn't like a valuable part of part of a skill set but one of the points you you make um and I'll say it poorly or paraphrase you and then you can expound on it um is that um whereas research is optimizing for accuracy and some form of ground ground truth um in business we want things that are useful a lot of the time these things there is sometimes an impedance mismatch here so maybe you can tell us a bit about what you mean by applied NLP thinking and why you even started to think about this um yeah so I think well we yeah we started thinking about it because yeah it's just something that kept kept coming up and it's also the kind of the area where we work in where people um have um a business problem that's often quite vague like we want to improve um the we want to make our internal I.T support team more efficient and reduce the time they spent um replying to requests and um you know there's a there's a natural machine learning or NLP tie in here but it's not a problem you could take one to one and um you know turn into a prediction problem like it's where you really need to think about what do I do how do I break it down and how do I reason about a what the system should do what makes sense and um what makes it useful and I think that's that's kind of what applied NLP thinking is about like really reasoning about um the best approach um for a real world applied contact and so I think in the post I also kind of compare the um these two like how and what um points so it's like you know the um this um you know the how like how to do a thing is stuff like oh um which you know implementing a machine learning model that's like of course something as an engineer it's important to know uh training knowing about ml more generally um knowing python knowing libraries and then there's kind of is the what like what should you actually be doing and that's kind of that's really the hard part because there's no easy answer to that there's no not even really you know there's no course you can take and that's not something you can you can learn because also it's also very specific like it really you really need to reason and that's like stuff like when should you even use machine learning in the first place yes um um and you know yeah like there are a lot of you know and often for example we tell people like hey if they have a problem start by doing a baseline that's just like really simple even just regulate expressions and like see how you're going because that's what you need to beat like if your machine learning model cannot beat that you probably you're either doing something wrong or you like it's really not a good use case for that because you're always going to end up with worse and less predictable results so um that's actually that's yeah that's also a big part not everything needs machine learning that's something you don't really see in research because it's not this is not interesting like you you know you have a you're faced with a problem and um you know with input data evaluation data and if you're trying to improve the accuracy score it's not an interesting academic contribution to say oh if I use regular Expressions I get this score but um similarly oh if I just change the data I get better scores like yeah sure but um that's not the point whereas if you're actually working in an applied field like sure change the data make your problem easier um that's absolutely allowed like this um um and yeah and then of course also understanding what does the application do what should it output and what are the outputs actually used for will the outputs be shown to an end user will the outputs be shown to a human who then makes a decision based on that what outputs do we actually need um how fine grained do they need to be and that can also then influence the choice of components um that are used to best solve this like I think there's often you know it makes sense like we often ask like humans we often reach for what's closest to what we can imagine um in our heads or we think of hey we have these support tickets for example and we want to know well what went wrong so a natural instinct is to say hey yes we will just labeling everything that went wrong as like a problem and then we're trying to see if a machine can predict that so at the end of it we can get out we can get out a representation from natural language support tickets to hear the things that went wrong and then we can analyze them further and send it to the right support person um that's for example that's one option but then again for example like Linguistics come in you start doing it you often start you need to start doing it you start labeling these things and you start realizing ah a lot of these things are actually don't even have clear boundaries like the computer doesn't turn on and the screen goes black or the screen goes black and it's all weird or something like you you start to like even applying your label scheme and your your your logic that you have come up with and you realize first hey this is really hard um yeah no clear boundaries and a machine will likely have a hard time with this as well and then even if you try training something you get the outputs and you realize they're all arbitrary phrases they all don't follow any consistent any structure sometimes you have screen goes black sometimes computer won't turn on sometimes um you just have a verb then what do you even do with this output yeah not very useful and so then you might think hey actually maybe I should try just predicting a category over the whole thing and use multiple categories like this is the I'm just using the example from the blog post but um they are like tons of similar um cases that will come up in all kinds of different use cases where yeah when we're thinking what's useful and not to use there like I when you say something like that I'm like maybe embeddings would be something that would be useful useful for that maybe not but that's something that we could could think about right and see what what brings most utility to a particular business question yeah and also and as you said before also the same model or the same trained artifact can be um really useful in one context and really terrible um or even harmful in another that's also why I think you know when we're talking about um things like you know metrics to measure um things like toxicity in language like this is of course this is very valuable if you have like a language model or your face you know you have these large models that produce arbitrary output and you want to put that in front of a human yes it's definitely you know important to check like hey what are the implications and what what kind of language does it output and what language what words does it know and how did it view those because yeah the impact there can be really bad but of course um you know having a low toxicity score that's that's not like a win or great thing in itself like if you are using these language model embeddings in A system that say helps your content moderators screen for potentially harmful comments and be quicker at like filtering them out of the queue um you want the model with very high toxicity because those are the comments um you're interested in and if you don't have that or even in your support tickets or email filters you want that because otherwise your model might um you know hallucinate something else um I've seen them yeah and I don't know your accounting department gets spammed with all the toxic emails you receive because the model hallucinates that they must be about accounting because um all the weird stuff was about accounting so they they can be like you know harmful outputs on the other side and you can't you know you cannot solve this with technology alone like you have to sit down and reason about it yeah yeah it's a question in some ways how us humans and machines work together yeah I can't remember who it was I think it may be Fred Turner um you see San Diego I think he was there when he said this that um we need to figure out how humans and machines work together and machines are very good at doing things that create problems for humans as well yeah well sure yeah yeah yeah yeah and then also I think the other aspect or even if we so if we surface something from a machine like a prediction to a human whether it's internal or external um adding that level of transparency like um you know Towing even if you you have like your internal analysts at your company um who then get suggestions from a model showcasing like hey this is suggested from a model this is um you know the probability this is maybe some background on why that's all important I think in a lot of applications there has been this like weird idea that oh you know humans don't want to uh or like you you should just pretend it's magic and like you know not not tell humans anything about it and I think that's that's wrong like even end users even customers and consumers uh can totally understand that and work with a system that has a machine learning component um that's transparent like you don't have to tell people oh this is just magic and the computer does it it's like nah this is a model that predicted this with 80 accuracy which is decent but um still means 20 of the time it's wrong exactly so I'm interested we've kind of circled around these things but I'm interested in what what from your perspective as a um software developer is what the biggest challenges currently in the NLP space are um yeah so we also yeah actually we do see a lot of that because we also have uh We've recently opened up um a Services offering against basically tailored pipelines and Spacey tailed analysis because yeah that was a direct response to that because a we also wanted to be closer again to real world problems because there's a lot that comes up and we kind of we want to be users of our own software in order to make the software better because I think that's often a problem if you are developing but as you said you built Prodigy originally in order to also to do yeah to do that but then also hey we stop we're using our software as basic Prodigy other things to solve problems for other companies and very specific use cases um if we've actually we actually have a really cool internal document where we list like things fixes and improvements I think we call them like collateral um uh um fixes or something like that came up during working using our own software for other people and realizing ah this is this is so inconvenient who wrote that ah [ __ ] us okay we need to know yeah um yeah yeah so that's kind of that's the that's the context so we are seeing a lot of it we do actually and one thing that I was um you know very excited by and I've also kind of confirmed our view there is that it's basically Talent analysis where we really offer to do the feasibility study and to help people like break down their problems tell them what to do tell them um you know analyze the data annotation scheme all of these uh parts that are really crucial and just really look at the project does it all make sense can it be solved yeah this has been very popular and I think that also speaks to um the idea that um or like the idea we've had for a long time that often the really hard part is deciding what to do and how to break down the problem and go into a machine learning pipeline like which there are lots of different components lots of different things what to even use in the first place and everything seems it seems kind of hard every thing um you know seems very involved or like even what should I it influences the data collection it's kind of this yeah this cycle where you have this idea come up with a label scheme based on that label the data realize ah maybe I need to frame this differently try again and um kind of breaking out of that and really getting this head start on how should I approach the problem is often um yeah often that the one of the hardest parts um and then of course like there are hard Parts along the way but once you know what to do and have figured like have confirmed that it works the training itself isn't even that difficult anymore like it's not you know that's that's usually not the bottleneck and then there's some experiments you can run to maybe improve accuracy slightly but um for a lot of these use cases well the majority of it will depend on what you're doing in the first place um uh how how good your data is and how good your evaluation is so um even that is like once once you have something that works improving it um is often quite feasible and then yeah often if people already have a business use case or have like something they're doing then even putting something in production um Can it can be doable um and at least the first step can work but um yeah getting over that initial Obstacle of like okay what what do I how do I start yeah that's absolutely that's a big that's something we see a lot of as well um so we did promise to chat about this and that we have had a bunch of questions and comments in in the chat around uh large language models yeah um so as we know we discussed this before I mean you know you can't my mum asks me about GPT and I if if my mom's asking me about something you know it's it's it's it's out there um no I'm kidding my mum's actually quite quite Savvy but a lot of my a lot of my friends are asking me about GPT and the future of Labor and stuff like that and yeah so I with all this hype um around larger language models I think you're in a fantastic position to help us reason through how large language models can actually deliver value in the world of NLP um so I'd be interested to know how you think through it and whether you're using any at the moment and what your thoughts are um yes so we've been yeah dude I could talk a bit about that later what we've been working on but um so in general of course it's you know this developments are um you know even aside aside from the hype and having to distinguish what's like what's hype and what isn't it's obviously very exciting to see these things work and it's also in a way it's something we've kind of been waiting for for a long time I remember be really early on when we started a company we were like okay back then it was like what you could do is sort of word vectors and maybe slightly better word vectors that was the level of like even just you know and if you're thinking of like oh information extraction like weddings and then like okay there must be right now we're not quite there yet but there must be a way to retrain more to pre-train more layers and really um you know have a way to embed more knowledge about uh the language and the word the world into a model because that's the big you know this calls that problem of like oh the model needs to learn everything from scratch that was usually really hard and then okay with the yeah Transformer models that became much more doable even just using the embeddings um that was exciting and even now having the having the large language models being able to um you know make use of that that's that's definitely exciting I think it's also a very good um even though yes um you know having your like um mothers and friends and grandparents ask about Chad GPT and then having to roll your eyes and be like yeah it's not it's not all like that um it's annoying it's for people in the industry it's great because um you know there's a A Renewed in