Revolutionizing Document Review with Generative AI

hello and welcome to the edrm global webinar Channel my name is Mary Mack I'm the edrm CEO and chief legal technologist today's webinar is a uh collaboration with our trusted partner purpose legal and the discussion is about revolutionizing document review with generative AI present and future our faculty experts are Jeff Johnson and Dove gold Medina we welcome your questions and feedback in the console they'll be answered either during the webinar or following the webinar and this webinar will be available for replay at your convenience for the next quarter as are all of our edrm webinars and we have some resources available for you I'm here Mary all right Kaylee walstead edm's Chief strategy officer is here with us and she will share what resarch ources are available today thanks Kaylee perfect I have been so looking forward to this great discussion today with two industry experts and we're loving our on24 platform with so many more ways for you our audience to engage if you check out your console to the right of the screen you'll see the question mark icon this is where you can type in your questions for today's faculty and we highly encourage you to do so slide will be made available after the webinar has concluded and the paperclip icon is for related content if you click on it you'll see today's resources carefully curated just for you you'll see the on24 engagement tools description for you to download to help you better engage with the platform there's a link to learn more about edm's fantastic trusted partner purpose legal and their Stellar Suite of services there's also a link to schedule a meeting with the experts at purpose to learn more about them and how they can assist you then up next on edm's webinar Channel tomorrow an exciting Florida clle event entitled my AI did it is no excuse for unethical or unprofessional conduct consideration of recent case law and ethics opinions with experts Ralph Losi and the honorable Judge Ralph tiger retired speaker bios can be clicked and popped up to learn more about today's faculty and then just like Zoom it's probably our favorite thing you'll see the smiley face emoji and if you click on it you'll see many more emojis to react to the webinar and we highly encourage you to engage with them we're having a lot of fun with them on this platform you can even give it a go now and you'll see the Twitter bird for social media you can follow edrm and purpose on LinkedIn back to you Mary thanks Kaylee and our faculty today Dove gold Medina is a SE a seasoned uh senior review manager at purpose legal where he brings a robust combination of technical Acumen and practical experience to the realm of managed reviews Dove honed his experience as a review manager at Epic where he tackled complex review projects and facilitated seamless collaboration across teams his foundational years as an IT technician at coxen company equipped him with a keen understanding of technological intricacies setting the stage for his subsequent accomplishments Dove is a reveal certified administrator holding additional certifications as a data analyst and in reviews as well as a relativity certified administrator and with his multifaceted background Dove stands at the intersection of technology and legal review ensuring Optical outcomes for his clients and Jeff Johnson Jeff is the chief Innovation officer at purpose legal has been a significant change agent in legal services and the technology industry with over 20 years of experience harnessing the Synergy of people and Technology Jeff has been pivotable in defining ecovery doc document review and Litigation Consulting Services Jeff's leadership has empowered some of the most the world's most premier corporations and law firms to achieve substantial Savings in enhanced efficiency in eisc Discovery his expertise in the development of machine learning Solutions their practical application and results validation within technology assisted review processes sets a standard for how technology from classific algorithms to generative AI can be used responsibly and effectively in the pursuit of justice and corporate accountability and without further Ado gentlemen Jeff please take it away thank you Mary um it's really hard in a bio to convey the the passion I have and the excitement I have for the use of technology in in in efficient ecovery review it's been um really the Keystone of my career for for a number of years now um working with analytics teams large and small on matters large and small and within service providers and for law law firms large and small really excited to spend some time with Dove and you all today with that let me dive into uh talking a little bit about our agenda and offering a couple of disclaimers right off right off the bat um I've got three disclaimers for you first we are specifically a addressing ecovery review there are lots of opportunities to apply Ai and generative AI within the legal landscape we are not trying to hit all of those we're focused on ecovery and ecovery review today um second if anybody logged in expecting an in-depth explanation of how generative AI works this isn't that webinar um we want to give a good practical understanding of how it works I doubt if you're going to hear or me trying to explain a Transformer or embedding or encoding or any of that today um but we want to give you a higher level practical understanding of these Technologies um finally these are quickly evolving topics we'll share opinions based on what we know and how we advise our clients today um I I fully expect that everyone won't necessarily agree with us and as the offerings change as pricing models change our opinions and recommendations will certainly evolve along with that with that out of the way our agenda for today again practical explanation of what gen generative AI is a mapping of that explanation to some of the solutions that are available for us today or will be in the coming months um as well as um factors you should consider in determining how if at all you should consider using these tools tools today or preparing to use them as as the months pass um we will try to keep some future facing content throughout um Dove as we dive in today I want to take a few minutes just to talk about um our branding of this webinar as revolutionizing ecovery AI yes so this graph that you're looking at talks about introduction of any technology be it a microwave or a cell phone um or something like the VCR or the DVD player and I mention the VCR and the DVD player specifically because the VCR was introduced in the late 70s but it took over 12 years before it sort of Le reached what is on this graph as late majority over 70% of homes had a VHS VCR in them interestingly the DVD player was introduced in the early 90s and within six years it hit that 70 70% Mark so some technologies have a faster uptake than others and one of the things we're going to talk about here is how quickly geni is going to have uh uptick some of the statistics you may see around will say that people are already using this at an extremely high level there's a lot of implementation going on we're not seeing people using it at those sort of late majority levels yet at this point it still looks like we're in an innovators phase where a small group of people are using it a lot of people are testing it and a lot of people are learning about it so when we look at this model again we're in in innovator phase um and most of the information you'll see out there will put you somewhere in that early place our experience is the same just anecdotally we have a few clients who have used it in small phases of review individual pieces not for a full-blown we're going to review everything Soup To Nuts with Gen um and I mention that because there is that fomo out there gen is all over the place with advertising with letting you know that it's out there um and we kind of wanted to let you know up front not everyone has all of a sudden like a switch flipped over to use that said the level of interest in it is a lot higher than what we saw say 105 years ago when the early predictive coding tar Technologies came around those it looked like it was going to be a slower uptick like I said VCRs took over 10 years to get implemented DVD went a lot faster we really think gen is going to be a faster uptick one of the reasons being is that it's not as much of a purpose-built utility something like Cal you don't use in your everyday life gen people are using it to write letters to write you know all sorts of other documents and and Jessica talk a little bit about the underlying technology but people are seeing it outside of the ecovery arena and because they're seeing it in their daily lives we really think it's going to drive adoption in ecovery so we do think this is going to be an evolution but it is going to be a quick one and um Jeff I'll turn over you to kind of talk a little bit more about that perfect that's fantastic thanks Dove just to follow up on a little bit of what you said you know 100% agree we see a lot of surveys candidly in my opinion it's not really even worth sharing any of the specific numbers in those surveys um I could gather the impression from those all published this year by the way that that either 4% of my peers are using these Technologies in all of their review or maybe 43% from a survey I saw I think it was about that earlier this week um honestly for a number of reasons I don't think those numbers are particularly useful or maybe not even intended to give the impression that everybody is using it I candidly think it's even far lower than 4% of of of reviews are using this technology extensively at least um I do think the surveys are useful in conveying Trends I think that for the surveys that publish the numbers from you know we this time versus 6 months ago versus 6 months before that we definitely see what you've said Dove that that these techn this technology is gaining interest and at least confidence that it will be part of this landscape a lot faster that than than some other Technologies might have have done before so excited about it but agree with you we are still in that innovator phase and candidly after all my years in the industry I think most of our legal teams like to be in that early majority maybe even late majority when it comes to adopting this these new technologies full boat um with that let me go ahead and and jump into a a conversation about um geni and the features that are available for us today um you know this this this landscape is evolving quickly um I think that six months from now this presentation would probably be different than it is today as of today though there's still still a lot of um at least from an advertising perspective a lot of smoke and mirrors and AI washing out there trying to um take advantage of the AI um popularity to to drive interest in Technologies and and that's all that's really not not a bad thing I candidly think it'll probably help Drive adoption of set some technologies that I've been advocating for years we'll talk more about that in a bit having said that there are absolutely real implementations of these Technologies um there are Standalone review platforms that already have significant implementations of of ji capabilities um for the most part the Standalone platforms that have a lot of these features embedded in them are not the major market share players those big players from a from a market share perspective in review platforms have all started to release their gen gen capabilities though those features are evolving um most of them candidly are still in limited availability that's not 100% true but largely it is with General release planned for later this year um as I think about the Gen capabilities within EC Discovery I think it's important to mention that it's not all the same it's not just either we have gen or we don't we can think about these features I I I put them in sort of three buckets um and that that's useful at least for today either I can take a document and summarize it or interrogate it I can take a document and get a review suggestion and maybe even do that as part of a mass operation to get review suggestions for many documents um I can possibly use gen to interrogate and get key data out of many documents at the same time and those are kind of the three buckets that we'll talk about today um which tools you have available for you really depends on the review platforms that you have available or can have available um so determining whether or not it's time for you to start using geni starts with what review platforms can you use what featur feates do they have already and is that feature available to you specifically as opposed to some sort of limited release I won't mention cost here we're going to talk more about cost in a bit but cost is part of that equation too um and can be a significant piece that I want to spend a few minutes just talking about gen and I will ask um some of this is going to start off very simply simply um and my reason for I I will start off simply and then link it directly to those feature sets and I just think that's a useful um way to make sure we're all on the same page when I remember when llms first came out and we were talking about gen and and people were trying to offer explanations of what they do um I I I heard it's just a very sophisticated autocomplete and honestly I I it took me a little bit to figure that out just because it seems so much more sophisticated than what I see on my phone when I'm typing a sentence in a text message and it gives me some ideas about what the next words are but if we start at that level and then work our way up it really is interesting to see how they are the same thing if you take that autocomplete example where you're typing on your phone have a nice and rephrase that as a question if you think of it as you're actually providing some sort of an AI a question that says I've just typed have a nice what's the most likely next word that I'm going to want to type and we can all remember our our phone is is throwing out day night weekend whatever um so that's a really simplistic example now if I move over to just run some of these in actual chat GPT today I can see that if I type that question I'm not going to get a word or three words I'm going to get an explanation of what may be the like ly um most often used word is and then some other words that might be in there those other words that would have showed up in my autocomplete in my messenging app um and then an explanation of when you might use those now that's cool you can get a lot of context that might be helpful obviously we all know all of this stuff so this particular example isn't helpful other than for illustration um moving on to the next example um what I'm illustrating here is if we offer some additional context and some restrictions about what we want the answer to be chat GPT change its behavior I'm no longer getting all of that stuff it's taking the context I gave it and the requirement and and giving us in this case I want a one-word answer what's the right word to complete that sentence obviously again very simple but illustrating here um and what I'm really what I think is this is prompt engineering there are entire books and countless research studies about prompt engineering um for today I think a couple of things are important one as we get into some of the ecovery examples this concept of prompt engineering is often the or one of the biggest areas of secret sauce for the different technologies that we use um it seems minimal here but this is important um for the technology and really it's it's really just offering clarifications possibly instructions additional information for the LM to use as it crafts correct and consistent responses um moving on to actually transitioning into some from that generic autocomplete function to something that we could see as a parallel to an actual ecovery problem to solve I do want to make a couple of disclaimers here right off the bat um I'm using this this example is me using retail GPT with a publicly available document just a patent document just never do this with sensitive data and always always follow or confidential information and always follow your organization's um standards for doing this type of thing with chat GPT beyond that uh you'll see as we illustrate this you probably don't even want to wouldn't want to do this with chat GPT natively anyway getting back to the secret sauce that I mentioned a minute ago um this is just a document summary i' I've uploaded that publicly available patent file and said Give me a summary of what's in this patent and chat GPT gave me that um this is just an example you can see that there might be useful um within document review there are already implementations of this where within your review panel you can get a summary and and possibly use that to confirm hopefully not use that all by itself to make a review decision but certainly could help confirm and and help with that human um decision process there are some tools that are that are taking this capability and really custom fitting it to specific use cases like deposition summaries for example I've seen some really cool demonstrations of of tools out there that that are developing this capability moving on to our next um example this is just a perhaps a more specific example of um document summarization where I'm asking a specific question of gen and it's giving an answer back who are the inventors of this patent gen is taking my question it's taking the information I gave it with the patent it's using everything it knows from its llm about how patent files are constructed and where that author information might be and giving me the the information back um do I'm just wondering as we've talked about summarization here do you have any thoughts thoughts on on how useful this sort of implementation might be and where we might use it yeah talk interrogation is something that is more than just uh you know responsiveness answer it can help you find information that might take additional time or another pass through a document very often e Discovery we create things like privilege logs and on a privileg log we very often have to indicate who's the author of a document so the same way there's a question Who are the inv enters you can ask who's the author sometimes you've got a document that says right on it by this person very often in E Discovery you have to dig a Little Deeper you might have to look at the metadata and see oh who's the author listed in the metadata sometimes that's correct very often we'll see the the the metadata indicate the author is the IT department or whoever installed whatever the word processing software is but this sort of use case gives you more options you can ask it give me one answer don't look at the metadata you can you know really get into uh some some fun ways to answer questions with document interrogation doing prlog is just one case and you know as Jeff said there's a lot of interesting things you can do with this and document interrogation really lets you quickly uh get information out the document cool thanks Tove moving on to our next example this I mean I I I included it as a separate category but really it's just sort of a very specialized use of document interrogation document summarization and and this is where we might think about starting to use these Technologies and candidly there's lots there are lots of poc's and lots of studies ongoing a few published about using gen to conduct an automate document review they are absolutely Technologies providing this capab ility today um using our little chat GPT simplified illustration um I'm really just providing a prompt that is saying here's the document here's what we're looking for again a very simplistic example is it relevant to that query how confident are you and then provide me with some rational and and decision for that confidence and rationale for the decision and confidence um and we can see chat GPT is telling me yes this document is relevant to your question I happen to know that's the correct answer so I'm I'm pleased to see that it's confident that's the correct answer and then it offers me rationale that seems appropriate um so you can imagine lots of um implementations for this and we will talk more during this call about how how we could use these for first pass review possibly second pass review you could imagine looking at incoming Productions with this tool pii review privilege review all of these things are potentials and there is reason well I'll talk more about that in a bit all of these things are great if it works so the does it work the question is really natively it doesn't at least not to a status that I would be happy with this exact example I ran it using Jack GPT um 20 times and of those 20 times 12 times it did say it was relevant eight times it didn't every answer was very they were it was three was the confidence score on every answer and every rationale seemed very logical and explained why it was or wasn't relevant depending on what it decided at that time um but that's not to say say that gen isn't useful and can't be accurate for document review chat GPT isn't a good source for it but we already talked about reasons why you wouldn't use chat GPT for this anyway um all of the production ready implementations for this involve an additional layer of of I call it secret sauce a lot of people do around that prompt engineering that I talked about a minute ago you're adding instructions you're adding clarification you're adding context through the prompt that will help this decision get responsive just as an example one of our partners I used their API to to run this exact same test and it was it got it got the correct relevant answer 100% of the time um so I was just illustrating the need for Innovative design around these Solutions you can't just plug them in to a need is every platform and expect them to the to work at the level we need them to AI has weaknesses um let's go ahead and move on with that I we're GNA spend a minute here getting into probably the most the furthest in the weeds that that we want to get in this in in this um presentation today my reasoning for that is is it's important um you might have heard people mentioned rag or responsive um or retrieval augmented generation and we we want to talk a little bit about what that means and why it's important um I'm talking about it here because it's incredibly relevant to that third bucket of features that I talked about where everything we've talked about so far has been I'm asking a question or asking for a summary about one specific document well another really cool feature that we can use gen for is I've got a population of documents or a search result with lots of documents in it and I want to ask questions of all of those documents well there are frankly Technical and economical reasons why it's not going to make sense or even be possible to throw an entire document population into a prompt and throw it to the to to the Gen large language model it just it's just not something we can or should try to do at this point that's where retrieval augment and generation comes in essentially we're introducing a preliminary step where we take the users's question turn it into a query ask that query of a non gen data source whether that's a text index some sort of other analytics index and identify key documents or key Snippets of documents from within the larger population that we want to submit as part of that prompt that's when the Gen gen would take over with that subset of the most relevant content and generate the answer that you need so um you know for a long time the saying garbage in garbage out has been common within analytics and it certainly applies here this is the differentiator that query piece and the the design of retrieval augmented generation varies from tool to Tool and that depending on any implementation of this that will change the responses that come back because they it will change the input that the um that the Gen gets so this even though some of these Technologies might be the same if I were to just you know early on I I had discussions with teams around this will be really easy geni is we can just put a chat bot into our into our um review platform and and ask a question and get answers back that's just really cool stuff it seems cool it is cool but it it seems really simple just based on that UI but the difference between a well-designed production ready implementation of that kind of tool and one that is just a chatbot slapped into a review PL review platform is dramatic that's another area where this this you know the secret sauce that makes these tools work within our context is is is really relevant here in in this piece so I I I mentioned this because because of this concept we can have technology within an ecovery review platform that lets us ask questions and get information across a search result across the entire population um so it's it's a really cool value ad and you could imagine lots of scenarios where this is useful um I just saw a webinar yesterday from a v a technology provider that has some really cool um features coming out around a very focused utilization of this kind of concept where the user asks a question the technology sort of expands that out into a bunch of questions um and and comes back with with a very nice presentation of key dates key timelines key entities um there's just a lot of cool stuff on the horizon for this particular implementation with that I think we're going to dive into um just some wrapping up of these features so one thing I do want to mention through all of these feature sets um hopefully none of it seemed like it was an easy easy button everything happens automatically any Reliance on these tools the document summaries the trying to do a mass document review um interrogating a population of documents the importance of the human in the loop to to properly oversee and and make sure the end result is good even given all of the Fantastic automation that's that that's already available or soon we will be available to us this cannot be underestimated it's absolutely necessary and as we develop our workflows around this tool every single use case deserves some thought about what's the risk if if the answer isn't 100% correct and what steps are we going to take to make sure that it is and and and I recognize 100% correct in all those contexts is not a possibility just sort of came out of my mouth but correct enough to accompany the risk level that that that that um accompanies any use case that we're talking about beyond the general summary interrogation and review capabilities I think it's worth mentioning that there are differentiators within these tools I've put them in just general buckets but within those buckets there are going to be different implementations of how those tools handle different languages how they might handle or not handle image files how they approach audio files there is really cool ability to translate to and from languages such that within an AI context it's possible for me to submit a question in English that gets used against a document population that has a lot of Spanish or Chinese content content in it but that content gets considered and is part of the answer that comes back to me and that answer still comes back to me in English um there's just some really cool capabilities there images the same thing I can I've seen some really fantastic approach to handwriting and you know just really translating handwriting into text that can be used in these prompts and questions as well as transcribing audio um and using that in the in the question and answer process um and then Al also a lot of the Standalone platforms that have um implemented a lot of the Genai capabilities they've already integrated this in really innovative ways with other analytics and search um features so there's just a lot to consider as you're looking at these Technologies which one fits your use cases um pricing is something worth spending a little bit of time talking about for most of us well one I should mention a lot of the tools that are in in even limited release now soon to be General release the pricing isn't really finalized yet some of them we don't know what the pricing will be even though we can already start to use and test them um So Pro pricing is is evolving just like the technology itself a couple of aspects of that are common in pricing this technology are worth thinking about one it's often expressed as documents but it's really not a document price it's it oftentimes there's a a limit to how big the document can be I don't mean to imply that that's two or three pages and anything bigger than that is going to inchor additional charge but really once you're talking about 10 or 20 Pages depending on some depending on the technology it might be more um and again it's expressed as tokens but I'm using Pages just as a general guideline some of them are maybe as much as 30 or 40 pages but at some point there's going to be a break where if the document exceeds that size it's either not going to be processed or it will Cur incur additional charges every time it's submitted to the Gen um also an important note is this is often not a one-time charge so if I'm running a translation plug-in for example and I'm paying for that I translate a document I pay for that translation and then my translated text is sitting in relativity everybody on my team can use it or I'm using some Legacy tar technology I pay to put the 100,000 documents in the population into that tar technology and then I can sort of use that tar technology in lots of different ways um without incurring additional fees oftentimes this is very different it could be the case that depending on the technology I submit a question I ask gen to summarize the document I get a summarization back some of the tools will actually allow us to store it within the review platform so we could see that summar ization um others when if Dove logs in and asks a different question of that same document the Gen is not going to typically use the summary that it created for me to answer Dove's question it might it will likely incur another charge for that same document in order to answer Dove's question if I'm using it for review for example and I decide I'm going to jump in I've put my um I've developed my prompts we'll talk more about how that prompting Works in a minute but I submit 100,000 documents for review I get my responsive designations back I figure out there was something wrong with my prompt I need to run it again I'm GNA pay again um so just something to keep in mind with pricing as as as we evaluate these tools it is not an insignificant piece of considering whether or not to use um these tools with that um Dove I think you want to talk a little bit more about how we Implement these and turn it over to you yeah so Jeff mentioned earlier on you know you can look at whatever platforms you're using now most likely almost all of them have some gen tools that they're either developing or might already be available there's also tools that are Standalone but that's a technology fit that you're going to need to really consider do you want to use the tool that is sort of an add-on to what you already have or do you want to use a different tool um this is something that I think is going to change over time a lot of the early imp implementations of predictive coding you might have your review platform but it didn't do the predictive coding you had to export a population out you did your training there then you took a result that you then loaded back into your review platform uh for Productions or F further steps so some of the J looks like that now over time more and more has gotten built into platforms so the tech technology fit that you are looking at today is something you really need to consider but it's also something you should consider is likely going to change over time but definitely you know you want to make sure before you implement something that it's going to work with your existing ecosystem you also have to look at your workflow you know there's a lot of phases to ecovery are you looking to push gen into your first level review into your priv review um and what can it do and where can you integrate it there Jeff said people are being really creative you know they are using it in ways that we may not normally be doing e Discovery now they might be offering a shortcut um and those shortcuts can be very helpful but you need to make sure that it's not going to um upend your workflow it's not going to uh cause any consternation that your team is going to have a smooth transition to using Ji and again as we said we're still early in this process right now is a really great time to test things out and to learn how it can work with your workflow and you should be flexible when you start out with it you might Envision one workflow um as you start using it you may see actually it's not doing this the way I thought it was going to do let me try doing it slightly differently and find a workflow that's going to fit for your team so that you can you know leverage this technology to help you in your process which comes to the next piece which is the cost as Jeff said right now the models are interesting and they are changing so that's something you should talk about with your providers talk about the costs um but also don't be too surprised if six months down the road their cost structure changes that's simply something we saw with early implementations of other Technologies um and something that we think may change here as well hopefully it's going to come down because right now the costs we're seeing um don't really offer the the benefits that we expect from technology and that we really want for technology so the cost is a real issue at this point in the implementation last piece that you never want to forget is responsible implementation this is something that often gets covered and I don't want to spend too much time on this a lot of people worry a lot about Security in the Gen models you know Jeff talked about you know you don't want to put anything that is sensitive into say chat gpts publicly available tools the custombuilt solutions they will all explain to you why the um security is not a major factor and it really is not something you need to worry too much about talk to your it teams talk to your data solution folks to make sure that they are checking off all the boxes and that they are vetting it um but I've yet to see an implementation that kind of failed along the way they've all done very very robust uh security implementation so you know the bigger pieces to my mind are going to be making it work for your particular workflow um and making sure that it's not going to be prohibit expensive as some of the limitations like they might be at this point which brings us to another piece of the cost which is it's not just running it it's making sure that the answers you get are good Jeff uh I think he he said that he misspoke when he said 100% because we in ecovery are not usually looking for 100% accuracy for any implementation be it humans doing linear review any tar technology we don't expect Perfection we always expect that there to be some percentage of Misses be it over Productions or under Productions same thing with Gen how do we ensure though that that those misses are within our defensible you know boundaries the way we would supervise human reviewers we're going to do QC and there's lots of QC methodology out there you could take a random sample you could do very targeted searches you could all you could you know look at your Search terms and say hey hey we've we've seen that of the 40 terms we used five of them generate 90% of responsiveness hits let's look at anything that hit on one of those Search terms but is tag not responsible let's do a real targeted look because those terms seem to be the key terms you can do the same thing with geni you can go in and you can say hey we had a data population we ran it it coded this many documents say responsive let's look at anything that is hitting on a key term that is T not responsive by the Gen why did it do that and and look at those to see if maybe you need to re-engineer prompt or adjust it the same way you adjust human reviewers how would we evaluate guar process we have illusion samples we have other tools that we use um that validate those processes we can do the same thing with Gen so we have a toolbox and I don't think we necessarily need to reinvent the wheel here so you know we can even use both I mean I've seen that in various tar contexts where you do an illusion sample and then do targeted s targeted searching on top of it we can do that with the geni we can look at it and one of the things Jeff talked about is you know some of these implementations they will generate a summary for you so you can go and you can look at a docum say like okay well the large language model said that this is a responsive document and here's its EXP ation and a lot of them will even have hyperlinks right in them so you can click on a couple of words in that rationale and go to where in the document that rationale is and you can look at the broader context did it get the context correct in marking this or if I go a paragraph up and a paragraph down maybe it's pulling things out of context and it's a Miss um again there's a lot of proof of context Concepts out there that are suggesting these processes are good they are hitting reasonable recall and precision that it can be defensible but you're going to want to check that yourself obviously because the same way you wouldn't just submit a brief from a first year associate uh to the court without checking the citations You're Gonna Want to check the Gen system as well um we've all seen the situations where a gen system hallucinated and got someone in trouble that's what this checking process is are you going to ACC see 100% of the documents no because then you've gotten no value out of running the model um but you're going to want to check some percentage of them the same you would any other process to ensure that it is understanding your prompt and that the prompts a good one which comes back to much of what Jeff has said about the human element in this um that's really what you're checking you're checking that the human element in this meshed well with the evaluation that the large language model did and we got a good result Jeff did you have any thoughts about this before we move on you're muted Jeff I apologize for that thank you not a lot I I I did appreciate that you actually hit on some things that I should have mentioned back when we were talking about rag some of the the the components of that is the ability to um request that the response you get back be limited to the additional context that the the the retrieval piece submitted to the process as well as even citations to that content which is where those hyperlinks might come from so those that's a concept I meant to hit on earlier I'm glad that you covered it here and I just wanted to hit on it a little bit more but other than that let's go ahead and move on um to kind of just talk about generous of AI in the context of tar um in in I'll explain why that that means both now and in the future and not just Legacy tar and I'll explain what I mean by that here in a minute um we kind of hit on this earlier I really believe that that we're sort of at a societal change relative to the trust and ability to to to feel good about using AI in our ecovery workflows and and the reason for that just just from a general daily life perspective I mean my I might be a little bit biased and unique just because of who I am and the people I associate with but I mean most people I know have chat GPT on their phone on their computer and they're using it often um I I was at a coffee shop with a friend just a couple of weeks ago talking about Ai and you know within a few minutes just some random guy from the table next to us came up to show us how excited he was that he's been using chat GPT on his phone and what he uses it for I just think it's it's out there it's in the air we breathe now in a way that Legacy tar never has been and that's going to increase Comfort I think that you know we might even squeeze some accelerated adoption up to whatever the adoption threshold is for those Legacy tar workflows I've seen surveys saying that that that the adoption of tar in 2023 just Legacy tar stuff grew like 12% I don't know how accurate that number is but I think from a trend perspective we just see that people are maybe they want to use this and they come to us saying look we want to use AI because we're hearing a lot about how great it is well there are reasons why you may not have available to you the Gen AI that you're seeing in the news but here are these other features that you may have not wanted to use them two years ago or five years ago but maybe you're more willing to use them now so I think that some of that adoption will increase um I don't want to open a a can of worms around family review but I think that as people consider um the possible use of gen to do document review it's generally not going to be doing family review so can we maybe apply Innovative processes to General Legacy tar workflows that that might gain some efficiencies too again I don't want to get too far into that can of worms today but it's an interesting topic we could discuss um that I think might come up as a result of this relative to gen AI specifically um I do think it's important at least for me I have a very difficult time seeing gen AI in a review process as anything other than tar tar is really a workflow the technology that drives the workflow can change and you know really when we talk about using gen to complete a review that is talk um Dove and I when we started talking about this webinar one of the things that Dove said is you know whether or not we adopt this really depends on is it costing us less is it giving us better results or is it giving us some other advantage that we can consider and and candidly as you think about completing a review today um you might see some advantage in cost then again you might not it's not an obvious question for sure or it's not an obvious answer for sure results I think it's it's really too early for me to say your re results are going to be better or worse I there are some studies that are starting to be published we aren't ready to publish any of our own research um in what I'm seeing in the studies that have been published is that again these are the studies I've seen are single case results they're not sort of we've done a study on 30 cases at least not recently and and gotten these results I'm not talking about over the course of many projects but on the on the studies that I have seen it does look like there's the potential for the results to be better um better recall although General tar gener tar gives us good enough recall today um over time what I've seen so far the Precision in those results Is Not Great certainly not better um but with with some additional prompt work the Precision could get better too so those results might get better where I think I get really excited about it is is I think there's the potential and I don't know how far out this is I don't think we're there today there's the potential to be able in one pass to do a responsiveness review a privilege review generate the basis of a priv log as well as document summaries all in one pass and when you as we start to develop the ability to do more of that then it's really going to be hard to question the value ad and the cost savings with a proc process like that again I don't think we're there yet but it is interesting to think um that we may be there in the future um with that I I will just mention as I said I think it's a it's basically a tar process so the QC I I will illustrate that you know early on I think it was the first webinar that I attended where a solution provider was was advocating for automating review with generative AI this was well over a year ago um and we he talked through the Fantastic results they presented it it was and continues to be a tool that I think is very interesting um and I had asked well do you have any test that you've run how are you verifying recall and precision a and um the answer to my question on the webinar was was basically or the way I heard it was well this is a different Paradigm you don't this is you're getting a rationale for every document and that Precision recall Paradigm is Legacy talk we don't need to think that way anymore I wholeheartedly agree disagree with that and disagreed with it at the time and honestly I suspect that if I were to ask that same person that same question today it would probably be a different answer this is a crazy time lots of things are evolving um I don't necessarily hold anybody accountable for an answer on this topic from a year ago um but I think that all the testing we use to verify um Legacy tar applications Legacy tar workflows um they apply the same way here as far as I'm concerned and we can develop the same statistics to demonstrate whether or not these tools are effective as we think they will be um I think we're running low on time and that's serendipitous because I think we're ready to to summarize and and and work our way um to the end here I think at a high level I think what we want to just convey is you're not missing out 40% of your colleagues are not running generative AI or using generative AI tools on the majority of their projects we're just not there yet I think the adoption will be a lot faster than it has been in years past and Innovation is going to continue to um Drive prices down and utility and use cases up and more broad um so that being the case I do think nothing in our presentation should sound like advis to you know stick your head back into the sand and don't worry about using this technology that's not our message at all um it just may or may not be the right time for you to do it today and that's really a case by case person Byers team by te um evaluation um that Dove anything to add yeah I think I said this earlier I think the role for J right now is as a great place for you to learn about the technology um that is where we're seeing it in most places um and in some cases people are saying like well let's test it and see how it does and maybe the results are good and you actually use it in a production situation but at this point most folks are going in to test it and to learn about it and like I said earlier as well you're going to have to figure out how it fits into your workflow because it's a new tool and the same way cow workflows adjusted the way review happened this is going to adjust the way review happens as well so um by all means dip your toe in the water don't put your head in the sand um but also don't feel left out thanks Dove I've put up our contact information I know Dove and I either one of us would be excited to um to to talk with anybody I know there are a few questions we had intended to save a little bit of time we didn't um I negle I also neglected to put on this slide you know follow purpose legal on LinkedIn um I've got a Subs series of our blog where I talk about these Concepts fairly regularly um and certainly would love to have all of you join me in that discussion and in future webinars Mary over to you thanks cheff well we want to thank our trusted partner purpose legal for making you and Dove available for this educational and thought-provoking discussion and our thanks to the edrm community for your kind attention if you enjoyed today's presentation show Dove and Jeff some love check out the resources at the bottom of your console find those emojis and if you really loved it don't forget to click the link to schedule a meeting with the experts at purpose legal and we'll see you next time on the edrm global webinar Channel thank you