Andreas Madsen on Independent Research and Interpretability

Key Takeaways

hey everyone um so uh i am here because uh jen and jonas asked me to say a few words at the beginning but i won't take too much of your time uh i'll be fairly brisk because it's really fun both speakers today as well as um just like this is the first in the series that jonas and jen have put together about mentorship and sharing so it's um a really fun kickoff um what i will do so andreas do you want to go to the next slide i know you're wrangling the size today yeah so welcome i uh lead cochlear 4ai so cohere 4eye is a non-profit research lab that's both focused on making progress and important machine learning questions we work on curiosity driven research as well as creating more points of entry into machine learning um and if we go to the next slide i think this will give a sense of what our goals are it's to change how where by whom research is done um and this is both by exploring progress and important ml problems but also by showing that good research can be done anywhere um and in the next slide really what you're seeing now is like some of our core programs so this is a community-led event um and it's also one of you know a seminar series so this is really such a cool event and there's also the research programs and we are just getting started we're at baby lab so we started a few months ago and we're a baby community so andreas if you go to the next slide yeah so today is super special because of that i think it's so exciting especially with the community for me it's just so important that the community is a space where we can throw events like this so starting the journey of changing how we collaborate and building new spaces and one of it's just emphasizing cross-institutional collaboration but it's also providing a space for researchers both with and without formal affiliations and providing entry points into machine learning research um so if we go to the next slide okay excellent so this is where i'll do my hand up and i'll i'll put the spotlight where i belong so this is a series that's been put together by both jen jonas and madeline has done an amazing job making this event come to life but this is one of the two series that have come out of the community right now um and that for me is really cool because it's a chance for us to shape programming as we go um so i will pass along to jen i believe uh i'll pass to you okay excellent got a subtle nod um and just have fun enjoy the event uh feel free to join our community you can reach out to madeline afterwards um or apply directly on our website but really excited to have andreas here because uh i've andreas i i believe is well actually i'm gonna stop there i'm gonna pass the gen who i presume will give you a sample of what's about to unfold but thanks so much um andreas if we could go to the next slide please thank you uh hi so my name is jen i'm a second year phd student uh in austria and a member of the cohere fourier community and along with jonas we are leading an interactive speaker series with the goal of kind of increasing access to research to people who maybe are outside of us a formal program like a phd and because our community is just getting started we thought a great way to kick it off is to have a bunch of talks from invited experts on ai but also just an independent and collaborative research such as this event and also to open it to the larger community uh and in addition specifically for the c4a community we also will have opportunities to actually talk to our speakers and get some feedback about research ideas some channels for informal mentorship and longer term we really want to support the formation of independent research project teams within the cohere for ai community and we're really excited to kick this off i think we have an amazing first speaker and we'll have events about once a month going forward and with that i'll pass it on to jonas to introduce andreas awesome uh thank you jen um we are very honored today to have andreas as our face speaker this is like the first event and so thank you for everyone who's also joining us today um andreas is a is a phd candidate and miller currently is researching interpretability and and for nlp uh primarily he's interested in focusing on ensuring interpretability methods and to provide valid explanations and today he will be discussing some lessons that he has on independent researching and um and open questions in in interpretability so if you would like and if you have questions uh while he's speaking you can prepare some questions and or if you have some uh comments also we will have a q a session later on so feel free to to get them ready um so i would like to pass it on to andres thank you um yes so my name is andreas and thanks a lot for the introduction um if i had known this would be the inaugural talk maybe i wouldn't have accepted but i just learned this um so i was an independent researcher in interpretability i still work in interpretability but as mentioned i am now a pc candidate miller um and so i thought it would be useful to talk about my journey to have sort of a 20 minute narrative about that and what i learned um and then after that we'll go into sort of the general themes and specific ideas of where you might want to work in interpretability and i should preface this by saying i mostly work in lp interpretability and so the parts about that is mostly going to be of our nlp interpersonality um so my journey at least with regard to machine learning uh started in 2011 i read this book collective intelligence and then i read it again and then a third time and about the end of the third time i decided that i was gonna do uh machine learning although it wasn't really the popular word at the time um and so in 2012 i decided to go to study applied mathematics for my bachelor well it looks like this or like this you prefer cakes and i finished that in 2015 and i knew i was gonna do the master that's like the default in denmark but in between that i had a little summit job um and this is kind of my big moment for explanation um and so this just was to make a prototype for a search engine so you put in your job description and out comes the through the magic of the surgeon then comes these resumes and there's many sources of bias here i was aware of this um however something interesting happened so um we were collaborating with this external headhunting company um and so we're presenting these different resumes and then they were saying okay what is great about them what is bad about them and learning through that process um and one day we're presenting a resume and this uh headhunter here they said i really like this one why because of the gender um and you might not be surprised about this but this is very illegal in denmark um and so one thing is to be biased another thing is to flat out a minute um and obviously they were a problem but a problem for me was that they would eventually become part of a feedback loop where they would interact with the search algorithm directly and it would learn from what they like and so now we have a bias feedback loop and there are many ways to disaster but this is a very very quick way of getting there um and so this was bothering me it was boiling everybody um and we didn't really know how to solve it we didn't even have the feedback loop implemented so it wasn't really the highest priority at the time um and it was just two months this uh job here for me but i kept thinking about this how was i gonna solve this um i thought about it for years and eventually i kind of realized that even given the most virtuous model as shining beacon of light the hopes to all mankind unbiased in all aspects um there's just nothing that prevents a human from filling according to loan bias so they can just go through the search results pick out the gender they like and that's kind of the end of that the model really doesn't matter that much um and also when i hit a wall like this is when i tend to find the most inspiration um and so the first lesson here you should always draw the humans in these kind of graphs so let's do that um and then i thought about well if this virtuous model doesn't really solve anything then what about the sexist model can that solve something um and what you can actually do is you could explain it right so the recruiter is represented by some parameters in the model and so if we can explain them those parameters in the model we are essentially explaining the recruiter and so we could go back to the recruiter and say well these are your biases you should rethink your thought process and this is kind of like a first step to a real improvement and i'm saying there isn't a room for virtuous models or ethical models you should absolutely deploy those but in terms of the explanation there's some real value here that you won't find anywhere else um this is interesting because usually i will say that explanation they will never solve the bias issue um they can lead to discovery and discovery can lead to change and so in this case well explaining this to the recruiter doesn't necessarily make them change the thought process right like that has to be something that comes from the discovery and that has to be their own decision but it is the first step um there are other motivations for interpretability this is kind of the paint painting i like to draw um we have the sort of the category of scientific understanding and model debugging this is very powerful in the industry because it leads to profit and and you shouldn't be ashamed of working in that direction um actually many of the methods which applies to that category of motivation also applies to the other category this is about ethics accountability safety and i'm not gonna really explain what these terms means because they are abstract um our hopefully through this example that i just talked about with the the recruiter um maybe you get some idea and so in 2017 i finished my master and for context then like you don't really do internships and you don't really do publications um you just study and and but in 2017 as well i then started becoming a freelancer in data analysis data visualization statistics machine learning things like that and this was great although as a freelancer you don't really get to work on very ambitious things because clients they don't want to pay for risk they want to make sure they get what they pay for and and so when you sort of have these discussions with your clients it ends up becoming very psychic learned solutions simple statistical models things that you know can work um and i like to work in the unknown i like not knowing that there is a solution and so i was never quite satisfied with this and i knew that to really work on sort of ambitious things where i didn't know if there was a solution i had to demonstrate that i could work in that kind of uncertainty right which typically you would do for publications um and so i ended up writing this distil publication uh although it was quite a journey to that happened in 2019 and if you read the affiliations it's going to say new form limited because i actually managed to get paid to do this publication but it is independent research there were no other offers than this um and so you might as well that is crazy like how did you manage to get paid to do this and the answer is you don't the conversation was that they were an outsourcing company and they wanted to establish themselves as a machine and capable company many other companies wanted this and my argument was well if you want to demonstrate that you can do machine learning you should do it in the public and what better stamp of approval than to actually get like a peer review publication however you don't like risk and there's a lot of risk in getting published um and so a zero risk solution here is that we just make a blog post and so in fact it does exist a blog post very similar to this article and it did bring them a new source of clients and so from that perspective that was a total success irrespectible of when it got published um and then i worked on on making sure this got published afterwards i wasn't paid for that um but most of the work um i had financed um and so there's some important lessons here which is there's no such thing as a toll failure like there might be a total failure but if there is you should you're working on the wrong thing uh you need to plan for these kind of intermediate successes so in this case writing the blog post there was the intermediate success right that was what prevented the total failure um there are other licenses you should apply your strengths so in my case um i was quite great at like data visualization web development communication and this was sort of the core strengths i needed for this publication um and because of this i didn't have to do fancy model development i could just take some simple lstm models they would train like an hour on a gpu and that was kind of it so it also required very few resources so you have to shape your independent resource ideas around your strengths and and using for your resources um and so the journey for this took about a year i started the work in march 2018 and then i got rejected rejected again and then rejected the third time these are editorial rejects which is something you can do with a journal so they would just have an editor read it and then reject it and they did give some feedback a lot of time and and eventually they decided that this was worthy of going to peer review so i got my reviews and i started incorporating the feedback i got for that and then march next year i got published and so this is i think pretty fast for journal publication uh i just did another one that took like two years um but it was very valuable for me um and so the lessons here is that you will get rejected and it will hurt uh there's still value in trying the publication after this i would not have been able to do if i hadn't had the opportunity to get rejected this many times er in such a short span of time and getting rejected hurts so you need to have a support network and no one can struggle alone and be productive is what i tend to say um your support network does not have to be federal researchers who is also trying to get published um it can also just be somebody who have ambitions in life and are struggling with that and before this i had mostly done experimental reports um which is i learned was quite a different writing style uh for papers readers really don't care about the details what are your learning rates what is your learning curve how did you split the data set stuff like that they don't care um they should care and it should belong in the appendix um although typically you won't even find it there another lesson i learned was that good enough is not good enough your reviewers might not find every mistake that you make but they will find a mistake if it's there um and one mistake can be enough to be rejected and so whenever you think or whenever i forge this is good enough um then that is really a moment for me to catch myself and think okay how can i improve this is there some assumption i can fight something in appendix i can write can i cite a paper that's going to clarify this and [Music] and the last thing i learned here is uh what i'd now like to call defensor writing versus there's going to be like one or two messages that a reviewer sorry that it's just critical to understand the paper but a reviewer might miss them or forget them not understand that they're critical and so i kind of learned to just litter these messages throughout the entire paper repeat them again and again and my technical writing i usually wouldn't repeat a message like that but in paper writing where you might have a lazy reviewer it is quite necessary i think um so while finishing this paper i had done freelancers for a long time i want this in vacation so i went to japan this is me finishing the paper and also went there because i was getting really tired of the competitiveness of the european and north american markets i wanted to know if i could work somewhere else um that was me at a job conference not for machine learning but nonetheless useful information and and it was also an opportunity for me to sort of get lost and forget a bit of our machine learning but wondering about was i gonna get home from this place out of nowhere and uh well when i did get home to denmark what would i do um and i kind of thought well this distilled publication was not going to be the last one and i think this was important for me to take some time to really like crystallize that fort for me because the next publication was a lot more difficult to write um without that sort of clarity of mind that i got by just taking some time off from it uh i don't think i would have been able to really complete that and that doesn't mean that like you absolutely have to commit is okay to say this is too much for me i want to stop um but i think you should take your time to really think about whether or not this is what you want to do and another thing i learned was that you need publications to get to nice labs um one to two publications and top nlp venues was the message i got my distilled publication and industry experience wasn't really enough and so this kind of settled it this still was not going to be the end and i started collaborating with this fellow called alexander and so alexander was a research assistant at the university where i started the technical university of denmark and as a research assistant he was part of teasing classes and they were trying to reproduce this paper and nobody could reproduce this it's a quite simple paper and so that's kind of strange he had other ideas for what we could work on but i thought this was interesting because it applied to my strengths uh it was very mathematical i started applying mathematics um it required a few resources because it was just these hundred neural networks so i could train it like in an hour on a cpu and background research takes a lot of time and this was basically the only paper you need to know um and so that really made that part a lot easier and the last part is that this was a talk i think at neuro ips 2018 um so there was a lot of interest in this but nobody really managed to do anything with it um and so i thought it was also a great branding opportunity and you should not be ashamed to think about branding because as an independent researcher i assume your goal is to not continue as an independent researcher is quite stressful um and so you should try and get the most out of the effort that you put in which means if you want to work in an area that nobody cares about you should probably not do that maybe you can do it later in life but independent research this might not be the time um so our collaboration went like this we met for one to two hours per week was a kind of classical split where i did most of the work um i presented the background in order to understand the challenges and then we worked on the problems together we thought about what kind of assumptions we are making how can we test them um how can we like split up the problem try to understand all the components um and then i would write down the actionables what i would do for the week and that was kind of how it went then i would come back with the results present the background challenges and on it repeats um and i think this separates a bit from how i see pc students intact with their supervisors um usually they just present the results and that's kind of it so just i would encourage you not to do that focus on what are the actual problems you want to talk about and an important lesson here for all researchers independent and art is to write down the assumptions and test them this is extremely obvious or actually doing it is very difficult and i think as you as an independent researcher you're going to be in a much better position to learn this skill will be much harder in the beginning because what i kind of see is that the supervisor like a professor would take over this role they're gonna do a lot of the assumption fighting think about how to test them if they're maybe a good professor um as an independent researcher you don't have that kind of support but you will be in a much better position to learn this and so try to always think about like is there some kind of assumption here and how would i test it um some other lessons i learned were that you can break the rules uh so one rule i really insisted on breaking here was that we were going to sort of deconstruct the previous paper and think about uh or sorry to explain why it didn't work and how we would fix this and then therefore why would our method work and typically you don't see this in papers what we could have done instead was just to present sort of the american fairy tale of um [Music] these are our method and this is how great our source is so that has like a much more positive tone um and we did get smacked for this by the reviewers uh but i also think ultimately this is what led to the spotlight award don't know this but that's my assumption and because what we kind of seen in papers after this is that they kind of adopted this kind of writing style and so it's sort of become its own little microclimate of papers which are much more critical than usual and then the next part is that you want to collaborate with somebody who's engaged i think many independent researchers who write to me they say talk about right working with a professor or something like that your professor is gonna have hundreds of papers already published they don't need another one um working with somebody who's actually engaged like alexander was in a similar situation as me he needed a breakthrough to too um i think was much more useful and the last part is that i couldn't really think about intermediate successes for this so instead i thought the part diversity i worked in other side projects i tried to uh publish an open source module that would make the visualization head for my digital paper for example and and i did get rejected from your hips and that hurt a lot that's my artistic pulse um and i can't really prepare you for that all i can say is that you're not alone and we did manage to do a workshop publication which is something i hadn't really thought about but this was we're good okay but this was a intermediate success and i think a very valuable one because um i actually did get contacted by a professor about becoming a resource assistant just because of this workshop publication and so i think usually if you talk to professors they're kind of discouraging workshop publications but they're actually very valuable and not that difficult to make and so you can think about that okay so this also meant that i did go to newer apps in 2019 because of the workshop publication um this cost a fortune if you were an independent researcher um fortunately for me i had earned quite a lot that year in uh has a freelancer and so i could deduct this and tax um still cost a fortune though uh you don't typically get any sponsorships and you have to pay the full industry price as an independent researcher the one thing i want to change here it is that but i don't have the power um and uh this was uh really valuable for me i had this moment here where i saw bing cam do a talk on interpretability and until this exact moment i did not know the word interpretability if you read the distilled publication i'm trying to invent all sorts of other words in order to describe the same thing um but all the professors i talked about with about explanations they were saying oh you shouldn't work in this area you should work on patient optimization that's the way forward um and so they were quite discouraging and i never really needed external support but um at this moment here i understood that you could become a researcher in interpretability and i went home and i read almost every paper that being kim has written and after that my mind was clear that our interpretability was something i wanted to do um which is funny because everything that donuts then was an introversible to me but um i just didn't understand that at the time um and i never really needed role models in my life but uh if i didn't need one this would be the case um and so being kim did do a talk recently a keynote that i cleared where she covers uh the high level aspects of interpretability and so if you're interested in this you should 100 watch this uh she does a much greater job than i could ever do um but that's okay because she needs to think about the high level aspects of interpretability i do not and so if you ever think about the imposter syndromes and things like that then also think about how your values might differ you'd have different resources you have different needs at the time um and so these kind of comparisons really are not valid um so with that is a great chance to go to the second act here which is about the general themes and some specific ideas um and so just to be very clear interpretability is the ability to explain it to percent in understandable terms to human um typically it's the model that we like to explain but it could also be the data set it could also be another human um there are many choices and so typically you have your model this is where 99.9 percent of the great money and the research goes into uh the rest of the great moaning goes to explanations and then we also have revelations which is how good are these explanations um and these three aspects are typically researched very separately uh in particular relation is very separate it's about is it useful to humans do humans find this valuable and does the explanation reflect them all and but i would encourage you i should say like these are very useful things to work on i work on this and the current resource is not that great although it's really increasing i think there's still a lot to be done but i would encourage you to think about these things how we can combine them so a classical thing to think about is how can i make a model that is easy to explain however what typically happens here is we come up with a very specific model which is really great at a very specific task and that is great academic research but i promise you they're never going to be adopted in the industry the industry is not going to have exactly those specific problems or specific tasks and so they're going to look for general purpose models and the explanations are just not a priority and so i would encourage you to instead think about how can i make really great explanations for one particular uh general purpose model like the bird model for example um instead and another way you can think about this is what kind of explanations can i make which are easy to evaluate and another kind of explanation you could think about is or another kind of research you could think about is how can i optimize the model with respect to the evaluation um we're good steven um okay so um these are true general themes and um i have also written a survey paper for you and this really is for you in the sense that this is the kind of story paper i wanted to read when i started um it goes through all of the different ways that you can communicate an explanation so each row here is um an explanation a way to communicate so for example input features is these words are important for making a prediction adversarial examples is you could change this word to this word and that will break the model things like that and there's one particular area here called class explanations uh so local explanations tries to explain a prediction or a single observation global explanation they try to explain the entire model but in between that there's class explanations where you try and explain an entire class and as you can see there's not a lot of work being done here but i think it's a really great compromise between these two levels of being very concrete or being very abstract and so for example we have concepts i think this was uh first shown by bing cam uh in computer vision it can be uh these stripes important for the classification of a stepper or natural language processing it might be that is an occupation important for predicting this pronoun um but you don't have to think about concepts you could come up with a completely different way of communicating and i think that would be amazing um another direction is because a lot of the work done in interpretability happens just a landmark okay um another thank you um another source of um okay i'm good um a lot of the work done in interpretability is in computer vision and computer vision is classification and i think this is because images are better or easier to visualize so it's a bit easier to sell papers in computer vision however when those methods are then adapted to nlp um they're also about classification right but we have a completely different category of models in nlp about sequential outputs or sequence to sequence models and so these tends to not really be explained not that much research on this and so you can think about how can these explanations be visualized what parts of the output should we consider um are there new ways of communicating and i think this is very relevant now that these big models like gpt models or something like that like they all signals the sequence models right and they need to be explained but there's not really any work on this so these are the four general themes uh i would encourage you to think about um but i also have specifics for you um and so in the survey um [Music] in the survey um for each way of communicating go through how does that look like what is sort of the general definition here but also what are the specific methods which might do this how do they look like and then finally what is future works that you might want to do here so for every sale examples this is about can i replace this word with this word is that going to break the model there's a lot of different permutations to consider the search space is way too big um and so how can we restrict the search space uh in order to look for explanations which are i'm online with what are we actually interested in for example is the gender of this paragraph important um so that could be one resource direction and so each row each chapter discusses sort of the very specifics of what you might want to work on here um and so with that um i think we'll move on to the panel discussions or q a awesome uh well andres there is a fantastic talk um thank you um so we'll go to the q a session and perhaps maybe we can start with people who want to ask directly so if you if you have a question what's your yeah the harding questions i can i can jump in with a quick question yes uh well thank you andres that was a great presentation um what's your sense on what's the state of the art in nlp explanation um let's say either for classification or because for generation it's like it's a nascent um research area for for generative models but on on the classification side would it be something like tcav or what's the top of the list in your mind do you think um um it's really not the right question to ask i think because you want to think about how do you communicate right so within each way of communicating to somebody there is different state of the arts um um [Music] i think maybe for input features the state of that would probably be sharp methods but they're also terribly expensive to compute uh for every cell examples probably like hot flip here might be the best uh influencing examples i might say represent the point of selections it's an incredibly simple method but somehow i didn't start working really well almost all of the time counterfactual explanation mice is a good method our apologies i'm sorry to say is really not worth that much natural language explanations all methods are terrible there's no state of the art um if you want to work on that i would encourage you just be very skeptical of what you read [Music] yeah i don't know if that much to say for us i think for linguistic information is also a field that you should be really really skeptical of there's only one good paper in this field which is about information theoretic probing [Music] yeah awesome thank you awesome um do we have any other question from um from the participants uh hi andreas yeah hi hi uh thanks for the talk uh you mentioned a point about uh we should think about branding and if the paper is getting a lot of interest sure like how do you decide if that paper has like a lot of interest is it the number of citations or is it yeah big names in the paper like deep mind google brain thing maybe i wouldn't look at it like big names because you're you're not going to be a big name so so correlating with that is not useful but you i would look like for example at what are how much select papers cited by others who are maybe not famous researchers right um and so like andrew trask at the time was not a famous researcher but i think after this he he got a research position at deepmind um although i guess some of the co-authors were quite known okay uh i'm just gonna let the organizers manage to the chat right yes yes yes there is a question that is about collaboration i guess i'm also curious about so you mentioned that you collaborated with one of the um one of the professors but they were at your university so research assistant research assistant yes um i i guess the way you were in the same institution um the question here is um what are the good places to to to find collaborations or to finding engaged collaborators um well i i managed to actually get in contact with alexander by just going to like a local every event for machine learning um and so i mean that would be a good place to start i think you could also look like who are the first offers of a paper you might like um maybe those could be people to collaborate with um yeah great um ander i'm not sure if i'm pronouncing that name right andrew is wondering um how do you apply your strengths if you haven't found them yet okay um andrew i understand perfectly um [Music] i think this is more of an internal search question like i'm sure you have strengths like that is not the question um just you might be looking at other researchers and you see the kind of work that they do and you think those are not my strengths and that might be true but if you for example think about that it's still publication right like there's a completely different set of strings that i used and a typical researcher would use right and so you can do research with alternative sets of strings that you have and in fact i would argue that you're able to do much more novel research by doing that right one more thing that i think one of the participants is asking is about um i guess credibility uh so he says maybe you could talk about common rejections and weaknesses that you faced uh maybe in peer review or what are the common criticism of your work that and and how has that changed over time i guess this is more a question of as an independent researcher right um something i had a hobby of doing was to read peer reviews from iclear because those are public and you can actually also read how the authors responded to that and you can even read reject papers i like to read rejected papers um and they're previews there i think you can learn a lot from that i think one thing i really learned was ablations studies um this was something that became very important for the iclear paper it did um and so i think we ended up writing like almost a 20 page appendix just in a blatant studies and so yeah that was definitely something i learned there also so if there's no other question i would say that would be a wrap um yeah can we go to the second to last light in the presentation so i can close this out yeah that one i think it's here all the way all the way oh okay yes that one okay one more yes all right uh so andreas thank you so much for the wonderful talk thank you to everybody uh for attending can we actually have like a round of applause for andreas thank you i know we're all on youth but maybe we're showing up on camera yes i see you but um i wanted to really quickly mention that next month at a time that we will announce including on twitter uh we'll have enough we'll follow up on this theme with an independent research panel that'll also look at a little bit of nlp rare languages and things like that so please stay tuned for that and now for the the general members thank you very much for attending