What Does Best in Class AI/ML Governance Look Like in Fin Services? // Charles Radclyffe // MLOps #2

Key Takeaways

let's go ahead and get rolling I wanted to just start by thanking everybody for coming with it it's great to see all these faces great to see all of you show it up I think it's a really cool talk we've got here with trials and to start out yeah this is just a quick update on what we're doing with the ML ops community we are we have a slack channel if you want to join us there if you're not already there we also have some forms and we'll show all of that we'll send a follow up with this recording so everybody who's here you'll get a recording of this and then all of the the links so that you can check that out but yeah - to start us off Charles can you just go ahead and can we maybe start from the beginning and talk about how how you got into data science and machine learning and what your journey has been with financial institutions sure so so hi everyone it's great to see it's going to be here do I do a kind of public speaking but it's it's a I've never done as many webinars as I have in the last few days and I'm sure that's the same already so yeah it's great it's great to be here and I guess the great benefit of of doing it in this format is you know we clicking people from a much much broader geography than we would normally be used to so welcome to everyone particularly those of you so yeah for me my journey began probably about about ten years ago I was I was involved in a business that was doing management reporting and and business intelligence and I joined that back company not really knowing much to be fair about about the subject and so I went out and spoke to clients and they asked what is it that what is it that you want from what is it that you need and it was it was in 2010 it was the moment where everyone was starting to talk about some daily science and and those of you who've been around in this in the sector since then will know that those days in particular you know as much or more so than than today the skill shortage in the industry was was really acute and so we were lucky we were in the right place in the right time our clients were mostly banks and financial institutions and so we said upon our missions who to hire really good talent in from from universities train them in data analytics techniques and tools and give them give them the tools and then we and we we got them into projects with with with large financial institutions we build a business in in London and New York doing doing specifically that and it was a great time we we sold the business way too early it's fair to say we sold in 2014 so there was plenty of plenty of runway still today in this industry but the founder of the company had had been through the financial crisis and so you know by the time that we were growing and stable and successful you know it was the moment where he wanted to cash out so I think you know this is probably a lesson for all of us now living in moments of crisis is you know it's very it's very grim times or can be vary in size but then when when you spot land sometimes you want to put some roots down when you when you find dry land and that's certainly what our founder wanted to do so I left I stayed stay with the business through the through the acquisition and handover and then I left I left a year or so later and it was a great great time and a great journey and you know I learned a lot from from our clients and from our team and it was just amazing timing because you know we were we were talking about you know in those days when I joined business intelligence and reports very quickly the conversation moved into analytics and then the conversation moved into what we called context intelligence which now everyone will know is natural language processing and we we talked about this journey from sort of data integration to business intelligence so from di to BI and it was quite clear that we were we were we were pointing the the arrow towards the future which would be AI and of the cenotes world that we live in we live in now so I've been quite blessed with that experience and since since running that company I've worked at Deutsche Bank as the head of technology for their Innovation Lab in London had a very short stint with a start-up which was a catastrophic failure that's another story and then and then I joined fidelity as the head of AI which was a wonderful wonderful role and I left six weeks ago on paternity and I'm not going back which means I'm free to talk to you all today about my experience so great to me her fate well yeah thanks for that I appreciate it I'm just wondering as I was looking over your your website I saw there that you are the what you would call a data philosopher and I was wondering what that is in your opinion and why you why you would consider yourself that sure so so again it comes back to when I was running my data science company and we were talking you know we're talking about data science and you know trying to educate people what that was and and I made a remark once today at a conference that we were running saying that the tech industry quite often uses very poor metaphors for describing what we do so you know the market ears pick these labels but they don't really they don't really describe what we do so you know people people my generation will have remembered the days where we were focused mostly on data warehousing and I just always felt there was such a poor metaphor for what actually the job of you know data warehousing architects you know warehousing in the real world is all about supply chain logistics you know an industry which everyone is acutely focusing on right now but data warehousing actually is very little to do without it so it's about you know storing data and having that kind of golden golden source golden immutable source for eternity and similarly data science isn't really there's a really science it's it's it's it's you know stats and hacking and coding and it's actually a lot of what those scientists do you know isn't that kind of experimentation it's a lot of you know data prep and other things and so the point I'm trying to make at that conference was you know we need to be a bit more careful with our metaphors and we need to be made less less less misleading as an industry and maybe we need people like me from a non tech background I'm not a techie bye-bye by trade I'm a I'm a lawyer of all things which is another story and we need people to be maybe asking the questions and and I said we maybe we need more data philosophers and and the name stuck that was seven or eight years ago and now I write a blog called daily philosopher which tries to call out some of some of the interesting things I come across in the tech industry perfect yeah I love that that name and so I guess my next question is just getting into this idea that the talk title let's let's kind of look at that a bit more what we wanted to go over with what does best-in-class AI and machine learning look like and governance look like in the in the financial sector and how do you feel that is portrayed these days yeah sure so I'll just talk a little bit about my experience with fidelity and you know for those of you who are working in in large enterprises then some of this might resonate you may be experiencing similar challenges or not and equally this might be interesting so so I was some I was hired because the feeling was they were doing a lot of work around putting structure around that automation program the RP a program which you know as we all know is very little to do with machine learning but the guy in charge of that was was continuously coming up against machine learning projects which were kind of side of desk activities on the most part what I would call passion projects from developers who interested because alien we have any real Commission for any real business cases for what they were doing and they also had a situation where you'd have you know senior people who might Commission something on a whim and so there's kind of two contrasting scenarios and there was a number of these projects and the risk was that if any of them failed and got leadership attention it could it could cause it's a disinvestment from from from other programs associated with with automation which which was a major program of work so what they wanted was they wanted some ones who come in and put first of all a strategy around ma I which which we're talking about machine learning specifically but we're using the AI title and and their strategy you know needed to help the organization make sure that it was getting a good return on its investment and then secondly to make sure that the right operating model like structure is in place to to deliver on that so that's what I was highly sensibly to do and you know one of the first things I did was try to first of all make sense as to what actually did the organization have in place and so you know that was a you know on the face of it quite an easy task because we we we spoke to the the teams or the data analytics teams and the people who you know actually do in the development and people who had been active and proactive on the internal blog and talking about what they were doing and we were able to uncover you know 15 20 projects you know off the bat but then it was quite clear that there was a lot of other activity which actually people were less forthcoming about not because you know they were up to bad things but simply because they didn't really have the mandate to do what they were doing they were experimenting with with all the right intentions on the side hoping to come up with something useful and hoping to get you know the kudos that come from this and so it became more of a challenge to really understand what what the organization had in place so we got to the point where we maybe I think we uncovered like 26 27 initiatives and I remember doing a briefing for the CTO and for his direct reports I said look you know here's a here's a map the imagine like a Venn diagram I've got all these little little circles I'm drawing them so now here's all the projects you know here's project one his project - his project three here's project 26 his allah project we're doing today his his the machine learning activity that's going on within the firm today and the problem is that you know here's another circle there's a imagine a Venn diagram with these two circles overlapping this is what we should be doing and of course there's an overlap there's an intersection between between those two and the question was you know we we need to go through an exercise to discover what what are the things that were really important to the firm but also what are the things that would be important but we're not doing yet that's that's the most important thing and all this other stuff is just crap good just stop it because we're using people's time you know for for you know not for the best purposes it's not know about people that aren't doing bad stuff they just you know they could be focused on way more valuable things so that became like the number one thing I think six weeks here and you know that was very clear from the CTO we had that mandate to to go out and really figure out what what should be in that place and then you know we were kind of I would say wading through the treacle two three four months later and we would we were dealt a gift and the gift came from the regulator and gifts never come from regulators those of you who work in financial services will learn that you trying to avoid regulatory conversations that always put a bad news but in this case actually we were we would had a gift and it was the in the UK we've got a financial conduct authority and the Bank of England they they basically perform different aspects of regulating the financial markets and they had teamed up in order to do what they could a survey of of the industry around the usage and application of machine learning and the reason this was a gift was because you know when you know someone like myself can can walk around holding a piece of regulation and say hey I need to do this because the regulator Islam because I'm being a bad guy and I'm being a pain in the ass so the regulator's telling me I need to do this suddenly it gives me permission to find out everything that's going on that that it was more difficult and so suddenly you know four months of like really struggling to understand the extent of all the projects within the next three weeks you know we had a list of 60 projects and and everyone was very forthcoming and it suddenly enabled us to get a visibility in see into things and particularly in terms of the risk and the controls that were in place or you know were not in place and so we submitted to the regulatory response and I signed off on that and and but also we realized that you know the the financial regulators were likely to to continue to do this on an annual basis because what they were concerned about was systemic risk within the financial industry you know what if lots the banks or doing some you know crazy thing with some some vendor solution that goes wrong and then the whole banking industry goes you know well you know we all know we had other risks we had in models you know it wasn't machine learning that was going to go wrong that was going to take out the financial industry but but you know we you know we can see very clearly that the FCA was going to ask this on a regular basis as if I gave us a second opportunity which is to say look you know let's get on the front foot now we know that we're gonna be asked this on an annual basis let's put in place some basic things to make sure that we're not just like causing a fire drill every single time we need to ask you for an inventory of our models and individual risk management so we put together some I would say loose requirements day one and and you know the results predictably was you know a lot of Excel spreadsheets for people to fill in and I said this to Luke the the CEO guy science a few months ago that's you know if you know some people measure their success in life by you know positive karma or something like that school some people maybe measure the success in life by carbon footprints you know do you have a negative carbon footprint if you are then when you die Eve you've been a net contributor to the species in my case I my success in life by you know how I eradicated more excel and I've created and unfortunately my time is fidelity you know I was a bad person because I created more excel than I got rid of because actually you know it's kind of nuts you know we're doing we're doing AI reporting in Excel it's it's it beggars belief um anyway that would that was the situation the best we could we could do and then I was lucky I went I'm up with Nick Emily in Bristol who's the illustrious leader of the Bristol FinTech so Bristol attack community and I was kind of catching it with him and had a coffee and filling him in about what I was doing at fidelity and he mentioned Luke and science and I said oh that sounds interesting Luke and then you know from penny drugs and that's why you know I think you guys are really amazing position because you know it's clear you know you need to have a platform anyone in this space is in an enterprise needs to have a platform to to to the very basic level to keep an inventory of the models that are being experimented on those four they're in pre-production and those are doing production you know all of us are using multiple platforms you know some people using you know zero mail people would be to stuff on AWS you know Jupiter notebooks you know whatever and you know you yes you can you know if you want if you're if you're if you're developing say with what's and yeah you might have some tools from IBM to keep a governance around what you're doing with with what's the studio but it doesn't help you for for what you're doing on other platforms and so you know people like me and my boss is one so have a single view on the organization and you know what is the extent of activity and what is the extent of risk that we carry and so like an inventory is like you know hiding it's like the basic level you know and until I met doc science the answer was Excel and from that basis then you know the next level is is then possible which is you know there could be some models that we develop which are like crazy risky and some models that we develop which are like you know who cares and you have different risk assessment frameworks for those and what you want to make sure is that you want to make sure that your developers are following those risk assessment frameworks and in fact you can you can you have that audit ability in place you know so people like me I just wanna see a report I when I see something we like you know read hopefully know read say we were green everywhere but I want to see where the red is so I can then take action and you know you guys do that which is which is awesome and then you know the other stuff figure out the stuff which you know it's probably more interesting maybe for the community about you know collaboration and Providence and the ability to to kind of shine a light on pee hacking and other stuff and you know give give you productivity tools you know that's gonna mean a second you know the basic stuff is just having some having some controls in place and what's really interesting just to kind of end on those pointers you know it's probably since about a year I've been saying you know there will be mandatory reporting on on this certainly in a financial sector and probably the timeline is 24 to 36 months so therefore those of us in the financial sector needs to kind of start preparing for that and you know get our together and then what's really interesting last month the European Commission published their white paper on governance of AI in the European Union and I know they've said the same thing and I'm just looking at the white paper now on my iPad it's like page 19 talks about you know the requirements to document what we're doing and have you know have that detail and that audit ability and good record-keeping in place and and the ability to be able to replicate the results of models maybe a year after they were originally developed you know all these things you know even the you know at a European level you know if this is a consultation paper it's not in law but you know it's a matter of time it's 12 months and so you know I think we're living in the in the 90s if we take the analogy of software development you know the kind of wild-west of you know write down code and email it between people and you know version control and all that kind of crazy that's maybe people always call be too young to remember but you know I'm just about to but you know we're living in those days of machine learning and it will all come crashing to an ends you know I think pretty soon and that's why we need to get a grip of governments so there's a long long rambled and so but I just wanted to give a bit of that's perfect yeah and and my follow-up question with that is do you think only with regulations will you see that change or do you feel that there's some places that are proactively do it yeah so I I'm not a I'm not a big believer in in regulation for regular series sake but I was so I'm not a believer in like wedding for the wheels to fall off before you before you take action so I think I think firms need to kind of you know decide their own risk appetite it's gonna depend on the industry you operate in I'll come back to the European Commission paper you know what they talk about is you know applications which are high risk or maybe activities which are high risk so you know if you only if you're in the business of autonomous vehicle developments then you know high risk industry yeah you're you're doing something that if it goes wrong people could die so so you know if you're in that space why wait for future regulations to come you know you need to have some basic controls you need to be thinking about your governments if you're in the space of like I don't know optimizing the network for zoom which is like super important right now because we're depending on it but if it breaks then you know no one's gonna die then do you need to have the same controls in place possibly not maybe you can be focused more on commercial activities and and and just you know getting getting done and not worrying about maybe documenting what you're doing maybe those things are like secondary so I think I think a common sense I would say a common sense approach you know those of you who are working in in places where you know the risk of what you do is higher then you know let's get some controls in place other things I think it's a bit more than nice to have but I think you know inevitably over time best practice will emerge and you know in software development now you know there's there are still people who are maybe more sloppy around their controls and documentation but that's a you know it's a very small domain of those people can operate in you know for the most part enterprises expects you know governance to be to be to be followed in best practices is widely recognized yeah that makes sense and I gotta ask how and you don't have to mention any names or anything but have you seen a catastrophe happened and what was it like how did that go over if you have and if not no worries we can move on to the next question well I think we're all like watching a slow a slow-moving train wreck happening but in terms of in terms of machine learning no I've seen I've seen plenty of train wrecks in terms of like traditional data analytics oh I could do one story which was a frightening one we we were doing some work for a credit card company is back in my my my data analytics company days and and they had a they had a customer with like clearly too much time on his hands and the customer had basically gone through every single credit card statement and build an Excel model and routes of the credit card company and said look guys you've overcharged me and then they didn't restate you know when you get those sort of letters you should like you know you should jump on them and they didn't they they basically timed out and so the guy went to the Ombudsman and said you know I had this problem I race a complaint they didn't respond look into it and of course you know then they looked into it and sure enough this car company wasn't actually calculating the fees accurately and and and the interesting thing is like how actually the fees compound over time it was like a rounding error but the railing error compounded in it it was it wasn't material in a sense of you know it was probably I can't remember now but it was like tens of pounds not not not thousands of pounds but the point was it's like your friggin credit card company you've got to get the right you know you have a fiduciary duty to your to your customers and so they the the regulator came in and said basically you know sort of out now and they put a penalty of a fine in place but the problem was they're kind of calculation engine was so you know was so deeply embedded in their systems I'm picking it took the months and we help them with analytics and we help them get out of that very difficult position and I think it was just a lesson you know there was nothing about that was a black box there was nothing about it which was you know we could um pick it it was it forward but I've often thought you know hell if that was a you know for similar suit scenario if that was something which actually you know the level of explain ability wasn't wasn't there then my god that could be not only buried for a much longer period of time but also when it did we need to come up it could be very difficult one pick and I think the interesting thing by that credit card company was they they were told by their by the regulator to write to all their customers and and tell them you know of the problem and so I think was a lot of customers that got like you know credits for to pound fifty on their next credit card bill and probably wondered why and you know we were we were the reason why ya know it reminds me of my my wife had been overcharged a few times by our phone company and she swears against this phone company one of the phone companies here where we live and I'm wondering if it might be one of those one of those problems that they were facing to uh uh-huh yeah who knows it but the I guess the next the next question that I had and anyone else who has questions feel free to put them in the chat and in about ten minutes we're gonna open it up so that if you want to ask questions directly go ahead it's but my my next question was more on the lines of do you feel the current situation right now with machine learning and AI making the predictions that they're making is there anything that is borderline dangerous happening oh well that's that's a great question so I think I think yeah I think you know I just stick to my stick to my lane and and that's that's that's financial services so I would say that the the I can't speak for fintax because they they kind of operate you know you know you know come you know by nature more opportunistic man because they're trying to exploit the fact that large financial institutions are slow moving and so they probably have a different risk appetite but I would say that large financial institutions of very very cautious about about what they do and I think you know my own experience in here there may be people in the industry who disagree with this statement but I think my own experience having worked in in financial services before the financial crisis and after the financial crisis is that you know it's it's it's it's a totally different space than when it was if you've seen like wolf of Wall Street you might get a sense of what it was like in the 80s and I you know when I joined in the early 2000s it kind of still was a little bit like that and you know what happened in those days was there were there were people who you're in the business lines to and their job is to make money make money for the bank or make money for the you know the insurance company or whatever and then there would be a compliance Department and the compliance Department was basically the you know people used to joke they say oh that's the departments of of uncommercial C or the Department of stopping you do things and anyway if something went wrong it was their problem it was very much of them and us culture and then the financial crisis hit you know it was it was a very big shock for everything federal and working in that space and and I think the biggest thing that everyone in the industry experienced even people like myself who worked in technology was there was a reputational head so you know we went from being masters of the universe which was kind of one of the quotes I think came out of flash boys or one of those books or movies so at that time you know went from being kind of really you know top of the tree to being despised by by many people because of the consequences of the financial crisis were so grim for ordinary people around the world and so I think that reputational hit I'm definitely experiencing you know going into a party and you know telling someone you work for a bank and they like look at you like you're scum you know it makes you think twice about how you behave and and so so I think that that is much more ingrained in the culture so I think you know things which are risky in financial services things like you know decision making on investments you know to the customer without a human in the loop you know that's something where you know the the big banks will move incredibly slowly and they won't be very innovative because they're terrified there's something going wrong obviously there's there's other things you know like a chatbot for example you know might be generating language which hasn't been actually approved you know one of the crazy things at the bank is if you want to speak on behalf of the organization you have to be approved but a lot of the messages that you give have to be scripted so if I was on his webinar as a as an employee of one of one of my previous employers you know I wouldn't be allowed to say any of this I would have to kind of give you stock answers and all the questions would have to be approved in advance you know so they're super risk-averse and obviously no chatbots if-then that the language is being generated on the fly much as I'm just talking yeah talking nonsense you know that would really panic a bank so so I think it's fair to say that financial services today is maybe overly paranoid about this stuff and we may be the needle needs to move back a little bit more so that's my own experience I don't think there are risky risky examples in trading there's lots of mechanisms in place and we're seeing that now with influential markets which are incredibly volatile you know when when the markets fall by a certain amount that the market trips out I'm not sure if it works the other way around when the market rises by a certain amount whether it's well I don't think it does because they rose quite well yesterday but any when they fall them out it trips out and so you know I think the engineering arounds and making sure there is not a systemic risk is pretty good in other industries I can't speak to I have that experience but I you know I do worry that actually we we do get overly focused on I know this is that go back to my philosopher blog I do way that we sometimes we get overly focused on the engineering and and making sure that the the quality of the work has been has been done well and not focused on the application and so you know things which which which have caused really big problems like let's say Cambridge analytic oh that wasn't a failure of you know it wasn't a bunch of bad engineers working there it was a bunch of super talented engineers doing things which were like morally questionable and and I think those are the problems those are the risks we have you know we've got we've got you know really great technology lots of people trying to exploit that technology for commercial gain and and Maeby's you know we need to put some controls around the applications in some place so that's why I think the risk is more broadly in financial services the risk is just you know being dinosaurs and acting slowly completely completely and I guess you you make the case for more data philosophers to there's nowhere needs to be it there needs to be more of that happening yeah one of me and seventeen of you in the school so well I just have one more question that I see people are putting questions in the chat so we can answer those I I watched your TED talk I thought it was great if anybody hasn't seen it check it out we'll send a link to that too and you made the case for taking the humans out of the loop in the agricultural sector in that TED talk and I'm wondering if you felt the same way about it in the banking sector and how far off of that dream you feel we are right ok so say so say so yeah where do I start with that so thanks for watching my TED talk it's it never did go viral on YouTube I was gutted this because I have known enough Peoria psyche I was to do with the T monster thank you for watching it and if you haven't watched please watch the point of my making their TED talk was the I was asked by the organizer so you talk about automation and technologies in a very positive way because there was a lot of fear as still as a lot of fear but you know when I was three years ago I did that there's no fear around automation and taking people's jobs and and you know something which people get very emotive about and and and where they asked me for is like working in that space and you know can you can you say something positive and uplifting it's like a it's a really hard task particularly you know dangerous ROI just go marketing kind of crazy or going well science fiction and so it struck me that the the problem with with automation is that the benefits could accrue to a small group of people I think we do see that you know an obvious example is say Blockbuster versus Netflix or Amazon Prime now as Netflix partially relevant so in the old days when we had video stores you had you know lots of entrepreneurs in every town who ran the franchise and so you had like these kind of millionaires got scattered all over all over the world but then when when Netflix happens you know the blockbuster franchise ended and yeah you end up with a very small number of people who have you know the billionaires and so that the pyramid it becomes you know really you know distort it and so that was that was my concern and and the answer that a lot of people give to solving this and it's one of the things that we're talking about now in addressing coronavirus is a universal basic income and my problem with that is again from from the lens of working in financial services is is I just don't understand how the credited industry would work and the credit industry would operates if you if you're only saw of income was was a government check how would you how would you borrow against that certainty it's actually like a feudal society and in fact I call it I call it techno feudalism in the in a tech talk and so I to you if your goal is to like help people survive after automation hits then you know ubi might might address that survival problem but it might not address the the other aspect which is like the purpose in people's lives and thriving as humans and and and also important things social mobility you know some of us work you know as hard as we can we we work so our knuckles bleed because we want to do better for ourselves and better for our kids and and how do you give those people opportunity if you just cut from a government check so those those are the things I was concerned by and so what I said was actually maybe this is going to sound like crazy socialist talk but I was talking about actually maybe we should invert the problem and and rather than worrying about good summation maybe we should proactively automate and maybe we should proactively automate those industries essential for our survival like free production so to take the anxiety away from people who are worried about where the next meal is going to come from and so you know might kind of view behind that is you know maybe we could tax those industries where it doesn't matter whether we automate and find those efficiencies like banking I mean does it really matter if your credit card company is a little bit more efficient and introduces a whole bunch of robots and lays off a bunch of people in maybe we should tax and disincentivize that maybe we should we should incentivize industries which are essential for our survival to automate super proactively so we can bring the cost of their products down and and and and people could never work live without without needing to work so three years ago I was worried about automation wrecking society and automation and the big thing though I worried about and I worry about today is social unrest I worry that you people are really dissatisfied with with the world around them and they feel that they don't have a voice in there don't they can't change anything they feel helpless then you know we see what what happens you know Hong Kong last year was a disaster France in Paris in particular you know the yellow vests movement you know social mess is a real really big issue and now we're living in you know these very difficult times you know I would say that's the biggest risk that we have today is exactly that so I think you know ubi ubi may may actually be a necessity now in the short term because a lot of people give me out of work a lot of people who need to eat pay their rents you know look after their kids and do very basic things we're probably going to hunker down in survival mode but long-term it's it's surely not a solution because it will only reinforce the the imbalance between the wealthy and the so so yes a long long and so explained in my TED talk but yeah the answer is we need more we need more automation in the things which were essential for our survival and and the automation that happens in other stuff like banking credit cards insurances is just a nice to have and those homes could probably pay more tax it's good thing I don't work for perfect well I see I see a question in here from JH I don't know if you want to go off of mute and ask the question yourself or you want me to just ask it Jay if you're out there feel free to jump on otherwise well I'll ask it in a sec here seems like he's I okay so Jay was that hilarious okay so here I'm okay yeah so I just want to ask a technical question about explainable AI or ml models is this kind of explanation so why models choose certain features why models behave in a certain way do models adapt to global topology of the feature space so for example for different markets the same model or all the same algorithm would need to come up with different kind of qualitative models do these characteristics of these models somehow reflect the topology the mathematical topology of the market that it's trained on all these kinds of things evaluated before an algorithm is put into production and if yes is this kind of evaluation mandatory or is this only for people who are scientists or data philosophers who have like a more in-depth approach to what they are doing is this mandatory for them or is this regulated if they do this are there any best practices so how do I tell about my own measurements in evaluation if this is done correctly and if I do all these kind of things do I also monitor my current model in the wild because I can only train the model on certain types of data on historic data datum may get skewed in reality over time or real data might actually not be applicable to the Train model really is this kind of behavior monitored and how do you go about it how do you how do you even monitor it because I think here even human must be in the loop in order to do the qualitative evaluation how do you do this if you do this sure okay it's a it's a it's a it's a great question we could probably spend the whole hour discussing this but let me give you a quick answer on this so I think so explain ability it's it's a funny way the philosopher me will argue well act we call it explain ability we should call it exploitability it's somehow the AI industry has invented a completely new word to describe this and I've never met anyone who could tell me where explain ability as a word comes from but it seems to be just just a word only we talked about in in the state of science community but it's it's it's an interesting one because I think a lot of software vendors pick up on a the non-technical person's paranoia about the black box the system not being able to explain its results and what they you know what people don't think about is actually where black boxes humans you know we if I crashed my car and you know someone the policeman stops me and says what happened you know I'm I'm not going through an audit of every single sensory experience I had and every thought process I was going through every neuron firing I just I post rationalize I say oh I saw cats running in front of the car and I swerved and I crashed or whatever lame excuse like anyways so you know we we post rationalize and we've built a kind of a system of trust around around that and it's good enough for supporting the world that we live in and so we don't get too hung up on this idea of explain ability with with humans but for somehow with machines we really do I think what I'm pleased about is you know so first for the to answer your question about regulation there's no regulation as far as I know on this so far but I think it's only a matter of time the European Commission white paper talks about replicability and I think that's a more important point is to be able to to run a model maybe 12 months later once you know you've upgraded tensorflow and you're running on a totally different data set and your models updated and be able to still go back and repeat the same thing with the same conditions I think that's more important and to be able to explain specifically why this this happened that's that's my personal view and it's good to see the European Commission reflect that in their consultation and I'm hoping over the next year industry will will nudge them towards replicability and you know having good best practice around audits of your data and your model version control and those sort of things cuz I think that's really important but what you do see and I I saw this from some of the big bigger vendors and I've got a huge amount of time for people like IBM particularly I think the outstanding organization but I got a little bit cross with when they keep coming in and presenting to execs oh you know this this governance tool can can help you select a different model which may be slightly less accurate but at least we can we can explain the results because I think actually you know in that in that space you know so not is this is an optimization problem you you want to optimize for performance of your model versus the explained ability of your model the explicable leti of your model and and actually I think there are other things we want to optimize which I think are more important then simply whether a human can can can explain exactly specifically why something occurred and I think those optimizations are things like energy consumption you know there were a lot of a lot of things where you know optimizing running running a model on on a massive data set and getting you know really really accurate prediction doesn't matter when something which which consumes a lot less energy can give an approximation which is a lot easier so I think we should be focusing on optimization of models on performance grounds but to simply focus and label and explain ability I think is the wrong the wrong vector and we should think about you know energy performance I would argue would be a higher a higher priority but it's something which the yeah it's something which I think a lot of software vendors like latching onto because they can they can they can scare people and you know and scare fear for yourselves so that's that's that's my that's my take on this but if you could if you like the basics in place if you've got good good controls or in your version control you and your data and you can wrap placate the results I don't think you need to worry too much whether you can explain the result perfect and I think I saw a question from Kieran if you're out there you want to go ahead and give it a shot honey can you hear me yes again yeah okay super I'm gonna have to fight slightly against my kind of stage anxiety because I don't want to just gobble out this message this question so backing up a bit from the explained ability earlier on in say machine learning pipeline you'll have to do a load of data cleansing and I'm from a software vendor who's providing solutions to financial services but there are little reluctance to do more than they have to when giving us projects and there are in data for us to work with so if we come across data when cleaning it up ready to train a model which just looks plain wrong I just wanted your two cents probably Charles cuz you work as a lawyer in this kind of sector is there any way we can hold those companies accountable for their data so we can give them a list of oh here's all of the the rows that we think you need to look into and maybe you need to mop it up and clear things up or is it better and easier for us just to kind of say we made these changes ourselves to your data based on common sense and it just looked wrong could work out what a practical solution would be for it and sell the model back to them but with kind of disclaimers underneath to say you know it's not gonna work in certain circumstances because the way we trained it was manipulated by us in order to just get it work because your data frankly sucked so yeah I guess which would be the better choice to go with like try and get these companies to become accountable and have more regulation so that if we find problems for their data they have to address it or is it better just to sell our product as is with the little disclaimer yeah okay okay great quite another great question it's awesome to be on a on a webinar like this where you know you get you get a proper challenge enough to get my brain working so thank you so when I saw this I think I think I experienced something very similar I won't mention the company's name but but we were doing we were doing some work where we were reliant on data that was coming from a third party it was it was a strange situation where actually that some of the data originated from us and we were we were you know we were publicizing and then that other firm was then aggregating data from us and our competitors know the people in the markets and then selling that as a bundle back to people like us so that we could you know do Ford analystics and what was interesting is that we we definitely had some data quality issues ourselves so the data that we were we were sending out in some cases might have been wrong but then some cases within we will be sending out that we got back was somehow transformed and and how it yeah how the integrity got lost through the process we have no idea so then we spent a lot of time not only dealing with customer there's calling inquiries but you know complaints really as to like why was the date where was the data gun where the data gone wrong but we were also doing a lot of data claim and to put some governance in place around this it was like it was a really difficult thing but we had a very open relationship with the defender and you know we were working with them and we were actually using some some tools that we were experimenting with also try and predict data quality issues so that we can be a little bit more open with with the vendor but it was it was clear that actually a lot of the value that existed in that in that in that chain was around the quality of the data there and actually doing the cleansing was actually adding value and the value wasn't going back to us it was going to code for the third party on terms of liability I I think it's a really big opportunity there and I think again coming back to the European white paper on on future regulation I definitely think there's going to be because the liability regime that we have in your moment focus on products and services and and fitness for purpose and and negligence and all of those things that we could have used to in our kind of in an old-fashioned world but I think when when a system malfunctions because the quality of the training data was wrong or it from malfunction in real time in run time because something went wrong with the data feeds and the data came from somebody else whose viability is that you know that's a really interesting problem for lawyers to make lots of money out of but I think us as an industry we've got to be really clear about that and you know I guess no surprise given this as a doc science hosted hosted cool you know I think a lot of the answers to this is just making sure that you've got the basics in place you've got your own audits capability in place you've got that data lineage under control you've got a versioning you know there's you know other brands of baked beans you can buy there's other brands of those but but you know these guys have got this you know out of the box and I think a lot of those anxieties about liability can fall away by just having something off the shelf that solves that so that would be my not so quick answer on that really difficult but important question Thanks good two cents on it I'm just wondering about the EU white paper is there like a certain term that I can look up for so data provider accountability for their data is have you got any key terms off the top of your head by concept for do you know what I'll let me let me let me put you on the spot sorry sure I'm actually I'm writing a I'm writing analysis of the white paper for somebody at the moment so I I'll do another review of it tonight or tomorrow and maybe I'll write something up on my blog or something and post it so you can get it that way I don't think it's something specifically on this but I think I think it's a really important point where where I think most people are focusing on is about and things like firmware updates so if you buy a product and then the firmware gets updated and then it misbehaves who's liable in that in that case and I think a lot of the anxiety or an IAT is around that question of it's no longer the physical product is the product plus the software I don't think I've seen anything asking about or what if the data comes from somewhere else and whose responsibility is it then because obviously this is a an ecosystem of capability but I know I've heard this question I mean let me I mean look at the paper again with that lens and if I find anything juicy that I think would be relevant or I'll blog about it that's brilliant thank you ever so much my question good great so it's seven o'clock where I am I guess the top of the hour I just want to open it up for one more question and then we can finish up is anybody else out there have anything that they want to say they want to ask yeah hi thanks for the thanks for that now any philosophical ideas on ml ups itself and this kind of new approach to this new ingredient in the mix no philosophical ideas I w