Production: Pencil's Down!

Key Takeaways

[Music] hello and welcome to another episode of ecovery chicks the podcast about all things ecovery and legal Tech hosted by two gals who are really really jazzed about today's guests anyways we're your hosts that's Bri and that's Angie and today we are continuing our edrm series we are now on production and Angie and I did do a recording of production a couple weeks ago but we really had to call in the experts so tonight we are joined by two special guests Nate Audie and Dan Elmos are here joining us both of them are with United Lux so I'd like to just take a moment to introduce them to you all Nate aie is a senior project manager at United Lex he specializes in E Discovery he has maybe one of the most diverse backgrounds I've I've ever run into in my ecovery career and that's the saying a lot actually because most people in eisc Discovery have a little bit of a rocky road before we found the good stuff he started his career off as a full-time martial arts instructor which everybody knows when you're performing at martial arts that the next thing you want to do is become a pastor so that's what Nate did but after a decade or so of working in the ministry and nonprofit organizations he found himself in need of a change and he stumbled into the world of ecovery which by the way Nate I want to hear that story one day after a couple martinis Nate has now worked in client services at United Lex for about four and a half years managing programs and projects across the board from Tiny claims to large scale mdls he's also been a bit of an intellectual sponge and tries to know a little bit of enough about everything in the ecovery world which I can say all of that last statement Nate is a huge understatement you are a Powerhouse when it comes to knowledge in the ecovery world and project management okay so that's Nate Danos also AKA Dano don't anybody say book them he is a season project manager specializing in a discovery also at United Lex he has had a distinguished career he's worked for several prominent vendors such as novatos data Discovery legal Advanced discovery which is now consilio and FiOS which was acquired by DTI I mean it's like maybe the better question is like have you not worked for someone Dan with over a decade of experience in the legal technology sector Dan excels in managing complex projects from inception to completion outside of work Dan is exploring the Pacific Northwest because he lives in beautiful Oregon with his wife and his two huskies and he often likes to travel sail and immerse himself in the natural beauty of the outdoors and he's drinking whiskey just for anybody who's taking notes Nate Dan thank you very much and before you say anything I do have to add because now I've done the business end of it but now I get to do the personal end of it in my life in ecovery I did have a brief Hiatus where I got to enjoy life at United Lex and I was a project manager never underestimate how hard the role of a project manager is that's one of the most [ __ ] hard jobs that are out there you're hurting cats your job is hurting cats not only that all your cats are pissed their claws are out they're ready to fight and they have extremely high expectations but that is a very hard job and it takes a very special person to be able to do it well and I will tell you that Nate and Dan both perform this job to amazing expertise and if I were anyone needing a project manager I would totally put my life in the hands of either one of these gentlemen the second thing I will say is you guys know how involved with women in eisc Discovery I am and how much I love to promote women in this industry well one of the things that's always been hard is to have men in the industry who are okay empowering women and I will tell you that both Nate and Dan on many occasions personally that I can attest to give themselves of their time they've given their wisdom they've given their expertise to me personally and to many other women and others in their workplace and I have a lot of respect for that so I really appreciate you guys taking the time to be here not only personally I feel honored that you're here because I have a lot of respect for you both as people and for your expertise but I appreciate you being here on the eisc Discovery chicks podcast because we're all about trying to educate people on ecovery and we want to hear from the people that really get it done every day who know what the hell they're talking about so appreciate you guys being here okay is that enough of an intro [ __ ] your business intro I want the personal one yeah I don't know if we can live up to that personal intro uh we'll definitely do our best and no I mean it's true every male dominated industry which is a lot of the tech world has this idea that makes women not as welcome which is awful because I will say most of the the best PMS that I've worked with Dan aside have been women not side cuz I think there's an amazing group of male PMS out there but U but absolutely want to empower women in Industries where they can do crazy awesome things all right well we have a an icebreaker question just because I know Dan loves Icebreaker questions he just got the slightest little frur of his brow like oh no what's coming we are excited about the Olympics in our household so here's your Icebreaker along the Olympics theme if you were given the physical ability to perform as an Olympian at any one summer sport what sport would you choose and why I I guess it's probably multi-layered so if it's if it's something I could maybe attain on my own in actual life take me back 12 15 years and I could look at Judo or karate or I did martial arts that was a world I lived in I was never anywhere near an Olympic level that's just laughable but I at least knew what I was supposed to be doing if we're talking about things that I have zero chance but it sure would be cool I would definitely go towards like lacrosse I absolutely love lacrosse specifically indoor lacrosse it's kind of like hockey but you don't have to not fall over on ice and and you're allowed to hit people in the head unlike in hockey which is fantastic indoor lacrosse would would probably be my my choice if if uh taking like actual abilities out of it all right well what about you Dan no I don't know I don't know how to kick anybody's ass so probably the track and field or something men's 5K or anything else and throwing a javelin or something would be kind of cool I don't know Angie would what would you do I'm going badman oh my God you are not going bad mitt I'm going bad mitt I will [ __ ] people up at bad mitt I will come I will come out of nowhere I love bad mitton I have not played in like 20 years I would not join if I could do anything in the Olympics though I would swim first of all swimmers have the most fantastic bodies so if I could be a swimmer Olympic swimmer that means I'm Fit Plus I would have no hair on my body so that would be amazing those are my priorities in life so Ryan locky the Olympic swimmer so he's now on a reality TV circuit and he has recently been in a show called the traitors which I don't know if anyone has seen Nate's making a face like you've seen it it's incredible incredible trash Allan coming incredible trash it's incredible trash heartwarming trash Allan coming in a castle in Scotland wearing fabulous outfits and people you're basically playing like a mer like a a murder mystery of sorts and Ryan locky is randomly in it and it's incredible interesting anyway turns out you can pivot from an Olympic swimming career to a castle in Scotland well you probably have to because you can't keep that [ __ ] for life I mean come on absolutely not yeah badminton though that's forever B's for Life B for life that's for Life absolutely shall we move this beautiful conversation on to ecoc Discovery I hate to do that because so much over for the sake of everyone we got to get this done before Dan finishes his glass of whiskey I have seen pictures of many a bottle I'm not concerned he's drinking whiskey from an official glass and there's no rocks or anything in that glass and he's drinking with such class like I almost want to put him in a tuxedo and give him a cigar right now just for those of you that absolutely do should we talk workflows oh yay that's a great transition yes yeah that was a hard left that's that was not a graceful transition we're in our first season guys give us a little bit of slack as everyone on this call knows making ecovery approachable and digestible is Central to our mission here at ecoc Discovery chicks so I want to start real basic like let's level set how and why would someone engage an ecovery service provider the how let's say you're starting from Ground Zero obviously your internet research is going to be where you want to start with the Google yeah I guess you start from there and then I would probably look at the biggest companies and then I would probably take it a step further and get into like LinkedIn and find out who works at these places if you know anyone and talk to them first yeah that's that's probably how I'd get involved if I didn't know at all how to find a vendor there are lots of them and they're not all the same why would you hire somebody the why is because you're working in the legal world I assume or you are legal World adjacent or you wouldn't be reaching out to us and you you don't know the part of ecovery that's that's the reason you're looking you should hire someone because our job is to do that part I do not practice law I never will thank God I have zero interest and it's the thing that I always say when people talk about the background and I'm like this is important to me but I will never code a document for you and I will never think about the content because that's not my place you're hiring an ecovery vendor because you want someone who knows how to do what you don't you go kick ass in the courtroom right do the things you need to do and let us make sure that your documents are treated right your your defens ability is high your forensics analysis if you need it or your at least your ability to say we know this is forensically sound it was collected the right way no one can argue with this that's why we're here so I think the why is exactly why you should hire a plumber to do your plumbing it's it's also a job within itself I mean you might think oh I could do this this is easy I just have to collect this data and you go down these rabbit holes and you find out there's a lot more to it it really is an entire different job than what you already have on your plate keeping everything defensible you don't want to spoil any data you want to collect everything and not miss things even when you are given data these companies will will help you find missing pieces too like if anything's been deleted there's date range gaps in your data so if you don't have the time to do it I would definitely recommend Outsourcing for sure yeah I feel like that's a big thing with companies these days like an internal legal department isn't built to handle the volume that a vendor is also and you don't want the liability either I mean let say you [ __ ] something up I mean that's on you not another yep you want to be able to turn around and sue somebody for that right yeah you want to look back and say you broke our contract you did it wrong yeah absolutely and Dan Dan's one that's really good at this too he I mean and you hit you just said this yourself Dan but just to elaborate on it you don't know what you're missing because you're not looking at it from the eyes of someone who's done this for a million years or who does this every day when you hire a vendor like a Dan or a Nate they're going to be like hey so why this Gap here or why are these label different or it seems like something's missing here they're going to point out things that you didn't even know was wrong and I can get guarantee you if Dan or Nate are asking you that question something's up and you need to go fix it right now and do it now before we get to production yeah that's such a big piece because once we get to production we're at the end you bring in that vendor at the beginning right at the point when you really need everything so we can walk you through to the end I've seen before where where they thought they had everything produced and you go back to the drawing board and you realize that you know you missed a few things like the searches didn't Encompass everything they need and it's sometimes too late to go back for these things so you really want to make sure that you hire someone who knows what they're doing well that is a beautiful segue into my next question we've titled this episode pencils down because we understand that that's an important concept for anyone involved in production so can you explain what pencils down means in this context and why it's so important basically pencils down means pencils down no more no more editing no more tinkering no more reviewing it's done books are closed so you can't pick the pencil back up mid process Midstream of a production you're changing coding redactions things are shifting in that production set and they're going to go out the way you don't want them to go out ship has sailed train has left the station exactly so pencil down for all parties everybody I mean you can't be adding documents to the set either you have to tag everything close it if everything in that Universe has to be done completed so Nate what can happen if people don't observe pencils down that's uh that's that's one of the most common things that happens and it's it's hard because most of the time the deadline we we did not set that deadline you did right you know we're working within your timeline but to give you the best possible actual production you need we need you to go pencils down so you know there's a couple layers of pencil down most of the time there's there's your Council pencils down and then if we have a review team support there's our review management pencils down and then it finally gets to the PM and production team where we're validating and we're looking to make sure things are consistent to make sure that you're not you know coding conflicted families and things like that and what can happen Midstream is if you adjust let's say you decide oh no we redacted too much out of this document or we didn't redact enough out of this document and you've already given it to us then you're going to get a production that contains mismatching information either that branding that information's going to get branded that way but the coding doesn't match and so during validations things things don't look right right or it's going to go out without redactions when you thought it had because you added stuff halfway and if you think you added it partway through and we don't know that you're now opening the door for privilege or pii going out the door that you thought was done but you never communicated to us you said I'm done with my my work you're good to go on the production and then you made a change and we're either then rerunning the production we're we're having to go into you know clawback we're we're looking at at fixing a problem that never had to exist in the first place and I think pencils down can be so cleanly done we work with production schedules with any any projects we can anyone that we're regularly we know we've got rolling Productions or we've got a planned deadline they say we need a production in two weeks then we automatically build out a production schedule and we say okay to get that production on this day of an estimated size we need you to give it to us at this time and that means you have to be done you have the hands off and you have to let it go or there's there's a lower and lower chance that you get that production on time you're now going to miss your court deadline because you were unwilling to give us the time we have to have to give you the product you need so what kinds of tinkering do you see people doing you know they give you something like all right this is ready for production and then they're like oh wait wait wait wait just one more thing what what are some common one more things coding changes coding validations privil PR withheld additional redactions you name it I mean taking docs out even or adding a new Doc oh I need to add this can you process this and add it can I have it today nope you just added an entire day to your to your deadline here it almost feels like production is like a a really big freight train and trying to stop it or divert course later on is challenging has major ramifications and is just like generally a disaster that's a analogy it's some you're right I'd rather have the train not go and be six hours late than to try to stop the train somewhere else and after that yeah yeah a good way to imagine it with those coding changes a lot of times its branding effect or its content effect and it will always take so much longer to correct than it will to just delay it is better to say we're not actually ready for pencils down than it is to say sure but have one Rogue reviewer in there changing a bunch of stuff yeah and when you do run into these situations where you are able to okay we can we can just pull this document out we can update the load file and correct it it's the same page count let's fix it that way sure you can do that once or twice but when you have clients with these bad habits your databases start to look like dog [ __ ] there are four different versions of a production a lot of them say underscore old or dnu do not use you just start looking at this thing six months down the road and you've got 20 30 40 60 80 100 Productions and then there's each one of them has modifications underscore R for we change read actions it it's not fun and and it's really hard to go back and and uh try to piece together what happened and why and which version they need to reproduce you know they'll ask for documents that was produced in this volume but there's like six different versions because they made so many changes instead of putting their damn pencils down yeah hold on because I'm coming it's gonna come full circle this is GNA seem weird at first so I restored a car and when I went to deal with the paint part of it like I wanted to do some Bodywork on it and then get it painted and I started like sanding down the paint and the car was silver when I bought it but before that it had been blue and before that it had been green and before that it had been orange and it took so much longer to sand it all the way down and do all the body work than if whoever had painted it before me had just taken it back down or delayed their thoughts like it took so much longer because you had to peel back so many layers of paint yep yeah and it opens confusion like Daniel said like when you're looking back six months from now the question which version went out and that should be the easiest answer in the world we should be able to tell you what you produced but I've even had situations where you know large Productions are now being used for another project for the or another project for the same client and we're having to transfer and then we're trying to compare to the coding decisions because it was produced in three different ways and one has redactions but one doesn't which one do they need do they need both are we really going to produce the same thing twice again for a different group when we're supposed to not do priv it gets so broken and it costs you money and I think that's the thing that should be the biggest flag and and it used to have a a council team member who who we nicknamed Scuba Steve I feel like it's fine if I say that because nobody knows who Scuba Steve is your name is Steve out there be you I already know where you're Scuba Steve Scuba Steve would ask us to produce and then review based upon the production deliverable we gave and no matter how many times we told them like no no no you need to make all of the decisions before we reran every production minimum of two times most of the time three or four and we talked to a bunch of times until we finally got on a call and just they're like okay we work for the same end client and you are making them pay three times and he's like what and it's like yes the same thing we told you before I got to get this into your head Scuba Steve like you need to understand you for five docks you just paid three times so it could be a cheap small production but it's not anymore because you've done it so many times take your time know what your production population is and look at the documents before we produce it I I think that's a great point that you both are making and something I honestly didn't have the the most appreciation for until I worked with both of you because it would be like hey we have this rolling production for this client and the production's been going on for a year say and you have all those instances like Dan was just referring to this is version five well where's version four well here's the one that was rejected and here's the one that wasn't well [ __ ] we've had turnover in our own organization we have our own recordkeeping but I wasn't here for the first you know three and then maybe Susie handled the next three and then Dan handled the next four and then now bre's involved and like what the hell is happening there's no consistency across the board and the client is the one like bleeding money at this point because every single time that we have to sit down and figure out what's the underscore Army was that produced you're paying money for Dan and Dan and Nate are expensive people because they're smart and they're really good at what they do you do not want to pay them more money per hour to help you I promise Nate I feel like you touched on something really important which is the value that service providers bring to the table you know your Scuba Steve never would have known that the downstream effect of what he was doing but for your expertise when you stop and think about it production is sort of small TimeWise compared to the rest of Discovery most law firms and corporations they also just don't do it on a daily basis so that I feel like is why an ecovery vendor is so important you've got an area of expertise and something that a lot of people don't have exposure to so can you guys tell me what are some areas of expertise where a vendor can really be a value add to their client absolutely as long as you find the right vendor this is what Dan talked on earlier right like there's there's a lot of vendors out there but the the right ecovery vendor should be able to support you and guide you through the entire edrm model the parts you know and the parts you don't and that's the whole point of the project manager and the vendor right like we have a ton of support departments and teams that do all the things that Dan and I can't but we know about them and we can help add value to every step we've got forensics and preservations teams that can help you with your legal holds if you have no idea how to put a legal hold how to collect data in the right way we have we have then the next layer when it comes to processing we've got full processing sets we're not just opening up emails and and looking at them in your own Outlook right which is is what you're doing or you're having an associate print off you know 20,000 pages of emails and you're you're flipping through them by hand right we can process those and provide all the accurate metadata we help you through the dock review process that's the part you're strong at if you work in the legal world you know the documents that's the part you know but we've got huge teams that can support you know reviewing two 300,000 documents in in a month basis or less because we can build out with appropriate people who we've vetted for you we can help you to produce this in the way that you present it to the court you PR present it to the plaintiff or to oppos in or defendant whoever you're trying to show your information is accurate to in a way that you could be confident it has what it needs to have we've got the workflows we've got the pieces and parts Dana you can talk to so many of the pieces and parts that we do on the PM side the thing I would look at first and foremost with the ecovery company would be if if they're trying to show you how many tools they have and how many bells and whistles and and all the the cool things they can do with technology smok and mirrors a second opinion because all those processes are done by actual humans like Nate who's a [ __ ] genius and can sit there and create a custom workflow but what happens if Nate wants to go have a baby or something I don't know month off we have enough Nate come on I was kidding knows this process is like the more simple and streamlined the company is in my opinion you're going to have a better better time organization is key I've been at places that really don't have many organized processes and workflows hard to find where things are at nothing's consistent it's just really hard when when it's that way so um talk to me about like how where can a a vendor add value just as far as qcing you know what's in talk to me about what's involved in that and why you're going to need a vendor if you think about a production that you've made we're looking at what you've provided and prepared to deliver somewhere you've spent weeks maybe months reviewing these documents you're not going to go back and look through not only every page but but all of the coding information and make sure everything aligns with what the production value is our QC layers both before we send it to our production team as well as we have a QA team that's their entire job they QA every aspect of the process they QA your data when it is processed they QA your data when it is produced they QA your data overlay that we need to make for something the goal of that is someone who has the knowledge of what it's supposed to look like is able to look at every step of the way and say okay we know we have this many documents that should be slip sheeet this way let's make sure that that matches that the is appropriate that the metadata is is removed or adjusted the way it's supposed to be because we're withholding this information those are things that you know on your end you can do in a small firm for a very small set of data but producing it in a standard ESI concordance DAT file if I said concordance and I said utf8 you're not going to know what that is unless you work in technology in some capacity those are the kinds of big ads that we can do is giving you the format that you need the format that's appropriate and correct and and having the right number of people the right teams to QC those steps along the way and make sure that you get what's right you can't QC either it's when people say Q your own work that's ridiculous you you can s like I made the damn thing of course it's right um and talking about QC I mean there's a very documented extensive QC process that's been tried and tested that value all in itself is huge and something that I honestly never had the appreciation for before I worked with you guys when you hire others to help you with this there's a damn good reason why you're paying the money that you're paying and there's a really great way of qcing this that's going to protect you and your vendor for that matter one thing I will say too is a lot of firms and in-house team just don't have the people power to be as elastic as you might need throughout the process some parts of this process are super times sensitive and heavy lift and so you're GNA have to scale that and like an in-house team or a law firm team you know they're probably just not going to be built for that because that's not what they're supposed to do so hire a vendor I mean this is one of those things that you can delegate out to experts that's okay because there's nothing that's subject matter expertise that's needed like Nate and Dan don't have to know what the lawsuit's about they don't have to know like why you marked things the way that you did that's not of importance to them we don't care I've had one client that I've actually said that to because they were trying to talk me through a decision way late at night and I was like I don't care I'm here for you if you have an issue but I don't care you make the decisions and then you tell me when it's ready and I've got you we will produce it the way you say but don't talk to me well you know I looked at the document and I'm just curious do you think this is privileged no I do not I don't know I have no idea what this is about I don't I don't want to know what this is about we care about your production we care about your work product but we we don't care about the reason behind that detail and subject matter expertise that you have and you're supposed to have I want to get into the hood because I kind of feel like we're at a mechanic shop picking apart production because it's a very small part of the whole edrm Spectrum but it's so complicated and production's kind of like one of those things like you don't really appreciate it until you're there the first thing I'd like to talk about are some of the time consuming aspects of production because I think this is a value for our listeners that might be reaching out to get help what should people know about how processing time affects production schedule more images more time the way you export images as well like you know tip files Port faster than searchable PDFs so depending on what types of images you're you're exporting as well I mean I don't think it's so much machine time unless it's a super big production that's really going to slow down the entire thing you know most Productions you could typically have exported in a couple hours but when you get into the gigantic sets it it could take a lot longer it depends on if you're talking about just that process of production the the combination of machine and Export or or if we're expanding to like the process of production which is is you you hand it over and and what happens now and and I've I've actually been asked by someone before they're like we're going to need this super fast we know you said 24 hours but we we know that the approximate can you just like save some stamping for us I've I've literally been asked and that to me showed that they have zero idea what they're talking about there's no save space it's not we don't tell the production team like hey I need like 20,000 Pages worth of machine time here for you do they think you're printing stickers like I don't so I'm pretty sure that they're they're envisioning the the the stamping where you would you know put your prefix and you'd roll page one and you'd flip and you'd roll page two you know stamping some of us did that Nate I mean I'm just saying did it don't do it that's the key right like did it once don't do it anymore but but the process also comes from when you hand it off we don't just turn around and get to kick it out we have to look at it we have to validate what needs natives what is images only what needs a slip sheeet what has native redactions right if you use a native redaction tool there's specific workflows to make sure that your native Excel doesn't give them a bunch of stuff you didn't want or we're looking at that native Excel and it has an embedded email that some weirdo shoved into an Excel and you want that embedded email withheld but you don't know that and so you just go and we look at it and we go well hey if you produce this Excel you're handing them that native you said is privileged and they go what and we're like okay let's show you how to redact that out of the Excel there's processes to all of those pieces before that machine time that Dana was talking about that can be quick but if we just push the button go you're going to get a sloppy mess we have to have the time to go through the process and again I talked earlier about like if you have a review team on our side as soon as you hand it off it doesn't even get to me yet it goes back to the review man who then gets to compare it to their coding and qc's and be like oh they downgraded 20% of the privilege we we've got to validate that now because it's our work product that they're saying is overprivileged or underprivileged right it's changed and and so all of those pieces add up and then after the quicker than most of the other steps branding and and exporting dependent upon the type of exports you're now getting into the QC steps that we talked about earlier so this next question I originally put it under the heading don't go chasing waterfalls but I think we're actually going to invert that and say do go chasing waterfalls what I mean by that is best practices are there things like what's your waterfall on every production is there a standard set of searches that you run before it goes out each workspace is different but a lot of the stuff will be the same like you're looking for privileged documents you're looking for documents that say they have redactions but might not you're looking for documents that are imaged aren't imaged or image incorrectly the goal is to tag the production with the universe and then click on each search and if there's documents in each of these searches it'll tell you there's something wrong with them like they're privileged and they need to be fixed or a lot of times down the road they'll make new tags Etc that don't get picked up in the searches you'll start messing things if you don't really kind of reverse engineer what the search is doing to double check and make sure that you're catching everything that you need to catch but that's that's what we refer to as the waterfall searches you touched on this a second ago and you said catch everything that you want to catch can you elaborate on that more what what are you trying to catch on the way out the door okay so there's different fields that our processing engine will will spit out if if there are comments hidden comments track changes Etc when those are in the search we know that those have to be imaged differently through our processing team so that we capture everything that they need to cut that's one example I mean there's many things like poly privileged withhold obviously like that's not going out in this production set you're just trying to catch issues before they get out the door everything yeah yeah absolutely the goal of a waterfall and I I love the phrase waterfall because because again I I didn't start in a in a tech-based world or a database based world in the same way but the idea is and and what you'll hear is the phrase a dynamic search and I like that as an idea because Dynamic versus static is really easy that goes outside of technology in the same way if it's static nothing's going to make it change we saved a list and it says this is what's there if it's Dynamic that means it it is automatically changing based on the information that you're coding and tagging and stuff are changing so we make Dynamic searches where there's there's like a parent search a top level search that has a tag that says this is our production population and then five 10 25 I have an mdl that has gosh darn probably near 75 I think actuallyy D has another mdl that has like 75 for a different one and it has more searches than you could ever imagine each one of those searches were built for a purpose it was built to make sure that if a document says redacted it actually contains redactions you didn't miss it in your review and so it actually has those or a document is tagged in one field privilege and this is what Dana was talking about what about after you go through pqc maybe the first layers of review you didn't have the same like second level privilege QC where you were downgrading from a privilege log and so now we're not running off of the same field that we were we're now working off of this privilege QC field that might be different and so we have to learn that and change those those Dynamic searches and pay attention and work you know with the hand off and your pencils down to know what Fields you've been using so we can properly identify any and every issue that might exist in your production population yeah and that's sometimes how you can tell pencils are not down when these seches start building up we're in more documents within them you're like wait a minute this is changing so you want to stop the freight train before it leaves the station you run these is you you make sure the train is in fact ready to depart in working with both Dan and Nate because of their expertise and production and other things there's expertly crafted searches that happen within the production world that you didn't even think about and they're just standard practice for a vendor because this is what they do every day so they're running these complex searches and their whole goal is to number one make sure that you're not missing anything that you think that you're you got it all covered but you had no idea that there's gaps there but number two like Dan said we really want to know if your real pencils are down we've talked about the waterfall or the don't go chasing waterfalls or should we Chase waterfalls or maybe I just want to get lost in a waterfall I don't know what other QC measures would you guys recommend other than those searches that we've already talked about I mean I think we we talked about those searches and we talked about you know kind of the the post production QC that happens after the export is there we've got a QA team that runs through and make sure the deliverable matches what we expected which is sadly often times where we do catch people not being pencils down if they did it mid production I had a I had a third-party pii review team that worked on a project that was known for it every single time we'd get the production to the finish line and then it would' be going through its final QC and they'd be like hey coding's changed and it was like no we don't care we actually got to the point where we got approval from the council team to ignore coding changes after a certain point point just ignore him we don't care what happened the other layer of QC is our handoff so once you give it to us and we run this waterfall we're going to come back to you with any questions we've got and and we expect you to answer those questions where your production will not happen and and I've had people say before hey why didn't we do this and I'll look back and I'll be like because you you never confirmed you changed anything sorry buddy but so we go back and forth we clear out all of these validation questions that might exist we then put together a document and we've got one that that our team uses and our our department uses and I imagine most vendors are going to have something similar that's a table that we can put into an email that tells you this is the prod volume this is how many documents it is this is how many pages it is here's how many redactions there are here's how many slip sheets there are here's the breakdown of the the coding validations that we see and then we send that off and we say hey here's a search to your production and here's here's all of these details we're good to go right we're confirming you with you one last time we've done all of these validations and we want your sign off to move forward with the production as we see it I think a lot of times when you were talking about the validation forms and and sending that information back to the client for their sign off if it's in a document I've noticed people don't like to open documents and read them they just kind of like to say it's good they said it's good it's good sometimes they don't look at these things so communication is key I mean a lot of times you need to just talk to them and ask them like for instance what what are you redacting and if if they're redacting like email headers you got to ask the deeper question do you want us to redact metadata now because you've redacted names from this file on the image like email addresses but they're still in the load file so you're still going to be able to find out who sent or received that document there's all kinds of little little things you got to think about and communicate with your client so that you don't let them fail I feel like that's such an important Point like people don't think about that you know you take something out you redact it and you're like good to go and people don't think that the metadata is not redacted and so that thing that you think you took out of your document that privileged information whatever that you think you took out it's still there or at least part of it is you know and some iteration of it is there and available for perusal by your enemies file names Etc I think metadata redactions are such a good point because I think it's one of the most under focused on portions of a review recognizing just because you took it off of what you can see doesn't mean you took it off of what can be searched and identified if we're doing a true uction and I think I think that's part of the issue is people who have been doing it for a long time especially if they work in smaller projects mostly they're used to doing it on their own and they're used to doing it in a program like Adobe where they are flattening everything into a PDF and they're just handdrawing redactions and you know Adobe has the ability to apply stamping at a certain level and they're sending over PDFs and that that doesn't match current you know ecovery standards that doesn't match ESI protocols for pretty much any I would say state but I would go to to country you know internationally that's not going to match what you need but in a lot of these small Court programs in a lot of these small kind of you know they're giving to to small firms back and forth they're so used to it not mattering that when they get to a bigger one they're like yeah whatever we're just doing a production and we're giving them 10 15 metadata fields that contain the appropriate information to show we collected it from the original SCE did you make that that same information that you drew off the of the image is actually gone lawyers we have a duty to provide a level of competence to our clients and so if you as the lawyer are not technologically a genius as far as production goes you know if you're not in the weeds with it you have an ethical duty to Outsource that to somebody who is so that clients documents that you're producing meet the standard of care full stop and so we we've had a common theme throughout this whole edrm series that Angie and I have been recording which is the ESI protocol as much as much as we all loathe the ESI protocol at times you know Nate just kind of hinted to the importance of that because that's one of the first things that a good project manager will ask for because they're concerned that you are meeting all the requirements that you've agreed to meet and that we're following the format that you've agreed to produce things in that's kind of like The Guiding Light here of of what we're trying to accomplish what are some common ESI protocol Provisions that folks should look out for and how can they impact the step of production here guys the best thing you can do for yourself in my opinion is get a vendor while you're still building out the ESI protocol and I think that that's contrary to what a lot of people do but the the best projects that I've had myself as well as one of our forensics experts if we're doing the collections as well as one of our review managers if we're doing the review we look over the ESI protocol and we can give you Insight based upon our opinions and that's what really helps you with catching those things that are major pitfalls in an ESI and I'll say outside of production the one that's near and dear to my heart because I have to yell at everybody about it right now is if if you ever have someone who is asking for what's called modern attachments or hyper links oh my go the r flag and and and it's not production specific it is the bane of my existence look look at the StubHub decision that was made in the last month greatest positive movement towards the hyperlink negation of relevancy as a modern attachment it is not an attachment based upon how prevalent and the way in which links are listed but that's a thing you don't know that's a thing that you as a lawyer hope hopefully you know you know that production Fields all those things I'll stop with with hyperlinks but you're not expected to know everything and I know that law schools generally are not really teaching ecovery to lawyers for the most part still they don't and it's which is scary a shame our job is to know it so Dano I'm gonna let you take everything about ESI protocols with Productions because I just want to say talk to us beforehand don't wait for production don't give us a finished ESI protocol reach out to us and say we have a produ project we're going to need come to us for consultancy because that's what we're here for can I yes and you and then flip it to Dan because I want to lose this thread depending on your jurisdiction courts have different views on this and they are more or less Draconian about it but if you agree to an ESI protocol that you cannot actually do like you don't have the technical technological capability to do it some courts say fine then you get sanctioned some courts kind of waffle about it like there have been some decisions out of California lately that have been a little bit calling that into question but you could get slapped for it to you Dan on that happy note no we we definitely see some weird ESI protocols coming through I just had one the other day when I came back had to make changes because they wanted a native production all Native that's fine first round was e dos no big deal second round I noticed we had just emails I'm like I'm not producing these natively you're crazy so I looked through again and it said yeah we can we can produce images but it's our last resort and I said well you're getting your last resort because these are these are emails and we're out they fought you on that they argued for sending their emails and you beat them down it was good job um yeah sometimes you just have to make the call on these things and and explain why and they'll usually work with you well talk more about that Dan because I'm sure we have people that are new to ecoc Discovery that are like well what's the problem with just producing my emails why Would we not want to do that well you don't want to accidentally open one and send it or modify it and change you know you're going to freak someone out if if you send a seven-year-old email you don't want to get into that situation so anything that has an external effect anything that has a link and or the ability to be sent is not something you want to open the world to access for and typically when you're told to give a native production it I in my opinion I think it's because these people don't know how to load Tiff images into a relational database they want to just get this file and open these documents and look at them and you know go from there you can always argue the fact like look you don't need you don't need emails we can send you PDFs they're just as easy to open you're still going to see the email you're just not going to accidentally send this to someone else so I mean this is sort of I'm going to take us on a little bit of a tangent here but I think it's really important both on the vendor side and the client side to ask why everyone is coming at this with a slightly different area of expertise and a slightly different background and a slightly different knowledge space take your pride out of it I feel like the more I learn the more comfortable I am with what I don't know right and this was really hard for me in my early career to ask why because you want to have all the answers and know everything but you should ask these questions ask the Baseline stuff like hey I don't understand what's the problem with sending this in a native format very valid questions and it's so much better to ask and temporarily feel like an idiot than it is to send the thing out and then really feel like an idiot yeah yeah I love the idea of of contacting you guys first and involving you in the ESI protocol creation phase because it is something that is highly overlooked and the reason why it's highly overlooked is usually legal council and paralegals and all people involved are like we just got sued and we're fighting about what keywords we're going to be running through this data and what's even at stake and who are the people involved and holy crap or how much money are we have to come out of our pocket with because we're responsible we're not responsible like what is our stance on this and do we have to talk to the media and do we have to do all like they're totally preoccupied with all these things that you need to consider at the onslaught of litigation that nobody's thinking about six months to a year from now we might be talking to our vendor about production and it might be good to make sure before we sign off on this ESI protocol that we can actually do what it says that we can do and I understand the Dilemma but you guys are absolutely right Consulting you up front front is definitely preferable because so many times you guys are saving the client from themselves the cost of your litigation can swing by millions of dollars depending on what is in your ESI protocol y like if you do not structure that in a way that is advantageous to the way that your data is stored and the data that you have it can literally cost you millions in attorney and vendor fees definitely bring people to the table and ask question question s being caught flat-footed being caught in a position where you have suddenly agreed to produce all this metadata or data and the only way you can do it is by manually reviewing three terabytes of documents I mean that's horrifying right that's millions of dollars right there feel like I'm really getting on my soap box side note I'm flat footed and I've never heard that saying before and now I'm like a little offended oh my God I'm so sorry right it's the right soapbox to get on Angie because it is pure and simple if you agree to not utilize Active Learning or to not utilize Search terms correctly or you have to manual review XYZ like you're agreeing to something you don't even know the context of because Discovery isn't there and the downstream effects of not setting up your ESI protocol correctly is detrimental and I would say Angie I want my lawyers to learn from you let's close this out on a fun note fun with a with an asterisk and several disclaimers because I want to talk about the big bad p word privilege so obviously when you run a production the goal is to avoid producing privileged ESI but here's my question to you Dan andate how often does privileged material get produced how does that happen and what can you do about it not often if it is often it's a weird thing or pencils weren't down like I say a coding changes we we'll let that happen if if it happens after the fact sometimes redactions are missed because there's a field that says document supposed to have redaction but they actually didn't and it slipped through the cracks that doesn't really happen though I I can't remember the last time bad privilege went out there are some situations where you might have a document that let's say for instance the processing didn't provide text and that didn't get picked up in processes along the way and that document goes out with the whole set and for some reason the word whatever didn't pop up and that privileged document went out it has to be something like that because we really do go through a lot of chat we have an entire QC team that goes through these things and looks for even common priv terms and I don't see it a lot yeah because we don't just look at your coding you know depending upon the project and things like that we've worked with you on priv screens and I think the big key word to what you said though also Dana was bad priv true obvious priv is pretty rare if you have a good review support team and you have a a vendor running your Productions it should be extremely rare borderline priv is where things get hard where it it's a decision made by reviewer and those are things that are outside of my context right that's like I said I will never code your documents thank the Lord but I know that those things happen because I watch priv QC workflows where things are being upgraded and downgraded and I know that there are borderline priv decisions being made all the time those I would say in a large production right like 10 15 documents not what we're talking about we're talking about a minimum of$ 500,000 doents over that the likelihood that there's something that is considered borderline privilege is hot but that's contextual and it was decided by whoever did that QC that it's private but that may be changed later and that's that's what gets scary is sometimes one person thinks it's this way another person thinks it this way both on the council team and later on they decide oh this should have been or shouldn't have been the proof screen should have changed so let's talk about what parties can do when that happens because obviously you're going to have some human error and making these privilege calls can be really complicated there everything is fact specific and there are a number of different kinds of privilege and the law of your jurisdiction might be changing or pick a factor that could affect this this does happen but what can parties do if they inadvertently produce privileged material typically you depending on what you produced you would talk about that with your client and they'll either go back and ask for that document to be clawed back and or provide a replace Ms if it's just one document and a redaction needed to be put in place we would basically fix that one document so that we don't have to provide a whole entire new set kind of sucks and that's where we get into the situation we talked about earlier where you you got a muddy workspace I that's probably the most common thing I see after production's already been delivered so what is a clawback can you break that down for our listeners basically where you go back and ask for a document to be deleted from the workspace including all metadata any network iterance of the document where they have it stored on a network as well so when you have a uction you don't just put it into the database it's also living on the network somewhere as it was delivered so you want to make sure that you get every piece of that privileged document removed from their systems and then you provide a replacement and they'll just basically overlay that into the production volume on their relational database couple things about that I want to kind of break down a little bit more so this this clawback that you've talked about this is typically an agreement between parties right that they'll negotiate before a production goes out we're not slopping a Band-Aid on after the fact this is prophylactic I'm sure that's most all agreements but even if you don't have that agreement I'm not sure the ethical standards I mean yall attorneys would know better than I do as far as how to work that out but with good faith I think you can probably say hey look we screwed up and you do it quickly in a timely manner they're gonna they're going to agree with you and say okay fine we'll take that out we peaked at it but we obviously can't use it now so yeah and and there are things like that yeah but like you can't produce something in to be super delator three years later be like Mulligan you know it doesn't it doesn't work like that a clag is not a small thing that's why the big faux PA like you said it's it's the big bad p word is to produce privilege I would say a lot of people love to blame their vendor for it especially if their vendor was the review team but there's a reason that we do so many samples and qc's along the way with the council team because even or review teams they still are humans and they are not the case expert like you are and so we have to have those QC checks we run those term highlight sets to help our review teams while noting that some documents don't highlight as well or that you know some documents if it's in the text it'll only highlight in the text because in the native it doesn't actually show and that makes people angry but it's a reality of Technology that's the reason we need you as the as the the subject matter expert to do those qcs along the way to flag oh we learned this firm is now you know we didn't realize they used to be their attorneys 10 years ago so we need to add them to the potentially privileged set well I just want to thank you Dan and Nate for being here to share your expertise because I know you guys take it for granted what you actually know and how valuable what you know is to other people that don't deal with production as one example but you guys of course deal with all aspects of ecovery on a daily basis we appreciate you and and your expertise tonight I want to ask you both in terms of production because that is our topic of the evening is there anything that you want to have as a parting thought or just a tip or trick or piece of advice that you'd like to leave with our listeners before we sign off I think communicate with your clients be preemptive pad your timing if you can and don't stress them out about their pencils down when you rush them they make mistakes even though they're done they're not done they they'll call you back at 300 in the morning and say wait wait you know take this out or add the doc or something there's a lot going on and you kind of got to let them relax a little bit so that they don't freak out on you and start making dumb mistakes but for your time if you can we always say 24 to 48 hour turnaround time on on you know standard sized production sets if I can tell a client's going to be in the weeds working through things late and kind of working from behind I'll tell him 48 hours I would say the biggest thing from all perspectives whether you're new to being a PM or at a vendor a firm you know your Ed Discovery counselor your a lawyer don't think of production as only being at the end the production is is it it's the final package it's everything you've been working for so you should be thinking about the production the day that you're collecting data you should be thinking about what is it we are hoping to have what is it we're expecting to have as we're reviewing what are the sizes and the timelines we're expecting to have if we know like the end is on a certain day don't plan on that production being finished that day that gives you no cushion of time plan on that production being two days or a week or two weeks depending upon the size so that when things aren't perfect because humans aren't perfect because looking at thousands if not tens of thousands and hundreds of thousands of documents is not perfect you give yourself the time for that change because you're looking and you're thinking about the deliverable starting at the very beginning I think those are great pieces of advice and guys I just want to thank you not only for being here tonight but thank you for being the dudes in the middle of the night somebody's like okay oh my God I just went through review for like the fifth time and we did all the things that we were supposed to do like a week ago and we were too busy to do and oh my god thank you for being patient with me H okay could you sacrifice your life to put this production on and it's like 2 am and you're having to put your family aside your your whatever your personal time aside and that's part of what it means to be a project manager and you guys are both great at it and it's a huge sacrifice so I just want to say thank you thank you so much for listening thank you Dan thank you Nate and listeners don't forget to subscribe wherever you get your podcasts give us a festar review and tell your friends family and co-workers about us and of course please hit us up on LinkedIn or at admin at ediscovery chicks.com with your thoughts questions and drink recommendations and before we go we want to remind our listeners to join us in September in Pasadena California Angie and I will be speaking in person at the techn security West Conference it'll be the week of September the 16th Andy and I are actually speaking on the 17th but you'll want to come hang with us before and after we look forward to meeting you in person please don't miss an opportunity for great ecovery content forensics cyber security and all the good stuff at techn security West in September until next time bye bye bye [Music]