TLDR Generation of Scientific Documents | ML Interview #1 with Isabel Cachola

Key Takeaways

just want to make mention here so it's an interview format you know something new here that we're experimenting with and like I said he said well Cachola decided that you know she accept our invitation and she was kind enough to do that and to hang hang with us a little bit here and answer some of the questions that you have been asking and I also put together some questions as well that I really want to ask her I'm very curious regarding that work she has put up so that work is the TLDR extreme summarization of scientific documents so he said well it's just an LP researcher a pre-doctoral young investigator Institute for ISO and that's a prestigious yeah in research lab so expect to learn a lot today and very exciting so with that being said I would like to invite several and just to introduce herself and just to give us a little bit background of the work that you do and what you're doing is current role hi all thank you so much for taking the time to sit down and talk with me about my work I'm very excited to talk with you all so like you mentioned I am a pre-doctoral young investigator at the Ellen Institute for AI I am originally from Texas I did my undergrad in mathematics at the University of Texas at Austin and yeah I'm super excited to be here awesome awesome thank you very much for being here and for taking the time with us so I guess you're working remotely right most of us are working remotely okay what time is it over there by the way it is 10:00 a.m. here named so I think I guess it's really good timing okay so what we're gonna do is the format I'm gonna use here so I had some questions that we put together together with with some of the folks that have attended paper reading discussion you know we had a paper reading discussion there on this paper so there are a few folks on the call that have been you know I've been discussing this paper prior to this cow so that's gonna be great because that way we can take some of their questions as well so I'm just gonna like all the questions and then just try to just try to get your insights and your input from this one okay is that okay yeah sounds good perfect so let me just I'm gonna just exits the screen here now and then I'm just gonna go and video mode okay right now we have a lot of people on the call so very excited for moving this forward okay great so let's see all right so we already kind of had some introductions right and I think a lot of the people that are on the call already know me a little I've been hosting a lot of like paper in discussions and I try to help the community try to understand some banner research and we're mostly right now focus on NLP but they're trying to move more into computer vision as well and come uncover some of those papers one of the ideas that came up is you know can we have one of the arteries attend one of these sessions and just to have a discussion I think that is that could provide really tremendous value for people that are interested in that particular research and so I mean I really really glad that you accepted it and if you're here with us today so thanks very much you know before we get started with the questions I'm happy to be here thank you so I guess one good question you know usually comes up especially when you're working on research or when you're going to start to work on a problem usually you know in my experience as well as a researcher what usually comes up is what's kind of the motivation for this research right so before you answer the question just I would just like to ask you to kind of these people some idea on you know what this research was about and what was the idea what was the hypothesis and what was kind of the motivation as well just the high level you really need to go into a lot of details yeah yeah so in our paper we introduced Tod our generation so a tldr is an extreme summary of a scientific paper usually between 15 to 13 tokens and lengths and it can conveys the main contributions or ideas that are presented in the paper so as far as the motivation we really drew a lot of inspiration from previous work and extreme summarization and scientific document summarization a lot of the previous work and scientific summarization has focused on long long summaries so think abstract generation there's also been a lot of work in extreme summarization but so far it hasn't been applied to the scientific domain so we thought this was a really interesting opportunity to explore extreme summarization and the scientific domain and furthermore you know staying up to date with scientific literature is an important part of a researchers workflow right and usually includes parsing long lists of papers from various sources reading abstracts and then trying to decide whether or not you want to read the paper in full until the ORS already exists as an important tool for science communication its presence in social media on websites like open review so we thought that trying to automatically generate these would be could be a useful part of the researchers workflow okay that's awesome I mean it's really glad to hear especially from you being one of the authors of the paper obviously I think that's what as a group that was kind of our understanding as well when we read through the paper by the way very clearly written we understood and that was one of the kind of motivations and and the contribution right so how do you take something like a paper and especially with the fast pace now FML and end up being particularly as well things are really moving fast and and we need some kind of tools only some kind of things that we need to use to allow us to you know keep you know it's tuned with all the research that's going on I think this is a great way to do it because you can provide kind of a really nice experience by providing people like TL DRS which I can use and this thinking use also to you know if you decide whether they want to go deeper into that paper so I guess that's a really great way to put it so you know really glad that this work came out now I think it's the perfect timing as well it can be a better time actually so great great to hear so I wonder I wonder getting to know into details like the concept of field er generation right how this came up like and you know feel the our generation I think is a really new task right I want to like get your insights here so how do you how do you would you define you know I really good TL DR when you're talking about TLDR generation is particularly when it has to do with in scientific context so what would you say is a good deal dr yeah that's a tough one in our in our paper we list six non exhaustive non destroying categories of information bits that might be included in a tldr so these were things like domain method or contribution etc so I think that a good TL DR would contain as much of these information bits as possible but in as short as we can possibly make it okay but I would I would say though that what qualifies as a good TL DR is going to be community specific and it's going to rely on some of the common sense within a community for what is considered to be common knowledge as opposed to details that are being introduced by the paper okay great that's that's a great answer actually really good answer so now I want to move on since we have an idea of what the TLDR generation is let's move on a little into the the actual you know how the research actually started I guess you were thinking a lot about data collection here because when I when I went to the paper there were really unique ideas here and that's why I think really added a lot of really interesting nests into this paper really new concepts on how you would collect this kind of data it's not an easy process right as you kind of explained in the paper but maybe you can give us kind of like an overview of you know what was the thought process when collecting this data set and kind of some of the challenges as well in the data collection yeah so the as I mentioned in the paper collecting data for Tod ours is not necessarily easy I mean in an ideal world the way we would do this is we would hand a bunch of researchers papers ask them to read the text in full and then bright teal yours but you can imagine this would be super expensive time consuming to collect so we when we were looking at alternative sources of data for this project we realized that a lot of peer reviews contained summaries about the papers that they're that they're reviewing and these are these peer review comments are written by domain experts who have read the paper in full and read the paper closely so we found that it was act that it is a doable task to rewrite these peer review comments into a tldr like summary and that that was certainly really exciting for us to realize this was would be a method of data collection yeah I could I mean in my experience I I could easily see this expanding also to other areas of research it necessarily just at the older generation but there are some really nice is that you know other fields in other areas of studies in borrow as well so that's that's I think is a really exciting part of the work as well just just to focus a little bit more on the you know the because I think one of the things that you all focused on was to try to gather a really high-quality data right and you know gathering high-quality data and that's the reason why in the process was kind of expensive and also maybe took that long we don't like we didn't saw those teachers in the paper along this process I'm really curious if you have any any information in that for how long the total roster yeah the process yes so I mean waste we started with doing the peer review comment rewrites ourselves to see how doable it was and then we asked a few colleagues to do the process and to sort of refine our annotation instructions before we started hiring out annotators I I would say this but the process in full took maybe three months estimate I I don't have an I don't have a number off the top of my head for like number of human hours but I mean I would say the biggest challenge was refining those annotation instructions to make sure that they were clear and we were getting high-quality TLD arse out of that once we sort of refined our instructions and also our training process the the ball got moving and started rolling a little faster what would you say was the like like the hardest part of the whole data collection biplane maybe the part that took the longest which one of those um yeah I would I would say that I was refining the instructions and also the training because we mentioned in our paper that we do provide an hour of training for the annotators yes so refining that training process to make sure that the annotators had a clear understanding of what the task was I probably took the longest yeah okay and you know really curious as well as well just to get some you know some information from you here regarding the like the ideal size of a field er I mean you even mentioned at the theology that quality with the LDR it's also a subjective thing right and that's something that you can't you know lay the rules and say okay this is what makes it a really perfect LPR but I think one of the things that was mentioned was the size of the TLDR so why would you say is like in terms of the of course the science and then those scientific papers but you say it's kind of like an ideal size of a tldr maybe maybe based on the analysis that you are conducted yeah so in our gold data set the average length of the gold shield er s was 20 tokens in length and we found that majority of the TLD Colt shield yards were between 15 and 30 tokens and legs so I would say around there it's what I would consider to be a ideal size of a gr yeah and and also like I really like the part that was reported on the different aspects of TLDR right it's not just about saying you know at the high level what the research is about and using those keywords but also being efficient or like what the novelties are and these how does a mole actually cut your all these different aspects of a few of the scientific paper and other scientific work so I think that's a quite challenging thing I would say that because obviously I'm kind of you know I'm kind of involved us in this process of generating field ers myself and this is a lot of manual work that we put into doing this kind of effort but we see the value there as well and you know that's that's something that comes up a lot I cannot really find the perfect recipe to generate the TLDR because whatever paper is gonna be different whatever domain it's gonna be different you know it also the other thing that was really interesting to the mention of you know having kind of expertise right if you have some kind of background and then the work that you are reading maybe it makes sense to just to focus on kind of what the details may be maybe low level details rather than doing at a high level and then you can also even skip right this was mentioned in the paper you can even skip some of the high level details as well just provide better the level details so can you can you comment on that what was the what was the thought process of deciding that you know so instead of focusing on maybe a high level picture there could also be cases where you maybe need some details so where did actually this idea came from when you did the research and analysis I'm curious about this so just to make sure I understand the question um specifically you're asking about like the idea of focusing on some of the contributions of the paper as a yes to like the more high level topics I guess in the paper yeah something like that yes yes so I think that when when as researchers when we're parsing through these lists of papers we sort of have we're assuming some level of background knowledge that the reader has when they're reading these teal DRS and this you know this isn't even necessarily an assumption that the paper makes right but in order to fit the main ideas of that paper in such a short amount of text we really have to focus on the what the paper is I'm contributing right what what is new about this paper yes and so I would say that that's the main motivation whereas for example like abstract you have a little bit more text you can maybe I've seen like many abstracts will go a little bit more into the backgrounds information on that paper right right great so no moving on from data no about the modeling part and that was also great thing also very curious as well on you know how do you all decided that you know you're going to go with the part so Bart is amol that was proposed for text summarization I guess if I'm not mistaken so you kind of adopted this and you can it did some fine-tuning as well on top of that and the experiments show that using different kind of input configurations you get very different results so that's seems really interesting but what I'm curious about is how was there like the model selected are they in the end right so how do you inside that this is the model that you want to use all right I mean I think that this is this decision was based primarily on the fact that Bart has had such great results in other summarization datasets and specifically it's the state of the art for abstract of summarization on exome which is the extreme summarization data set in the news domain so I think that that was the primary reasoning behind going with Bart okay great and you also in the modeling part also I saw some experiments around multitasking which I found really interesting as well and how multitasking was used in this particular research can you comment on the idea of the multitasking and how it would use just to give like the participants here an idea of artists use yes so the the way that we did this was we took and at the related task of title generation and we used an additional data set of paper title pairs and we appended control quotes to the source of the respective sources so we had a code for title for everything in that all the papers in the title dataset and we had a control code for tea of yours and this I filled the our dataset and then we shuffled these two datasets and then fine tune apart on this shuffled data and this allowed us to so so this did a few things our hypothesis was that because titles are short and they contain salient information about the paper that this would be a good related task to use and then the other part of that is that every paper has title titles are more easily accessible than Tod are so even though TL DR is a low resource task using titles sort of allowed us to augment our data with this related task okay great I'm also very curious as well and I actually wrote this question because I always want to do when I do conduct research with other teams as well look for those like hidden gems right like what was interesting about the research what would you say or because there's so much things that happen in the research so many interesting insights what would you say was the kind of the inside I was very surprising to you that you would didn't expect in this particular research um I was personally really excited when we got back our first round of annotations and bounce that this rewriting process was really doable because certifies you mentioned earlier this this can be applied to collecting data for Keo to yours but I think that there's other domains and other tests that this sort of thought process of take this text that contains a submarine for example of what your M task is going to be and then rewrite it as opposed to trying to write the TLDR from the full source the this allowed us to collect more data more more easily and I think that this was exciting to me okay great okay I thought also what that was a part of the paper that I kind of found really interesting as I'm I was thinking about how to Jenny and approach whatever you know parts of the paper whether it's like the data processing pipeline or whatever it is and I could easily see this being used like in different domains like for for instance news extraction generating some kind of news from an article and you know and give that same kind of value as well those type of readers I think it has really really a lot of types of applications and on that note I really want to ask you here it's something we discuss a lot in our paper reading these questions we always ask the question of how practical is this research write something that even as researchers we also think about when they're you know thinking about motivation and our contributions these sort of things what would you say are kind of like real-world applications of this research you know how can you be used in practice out there yeah oh my goodness I think there's so many applications that are possible I personally think the TLDR s can be used anywhere that you might find a list of papers so a good example is on search engines for Google Scholar semantics color for example usually whenever you look up you're looking at the search results the first it'll show the title and then a one or two sentences from the abstract but the first couple sentences an abstract aren't always necessarily informative to what the paper is about sometimes its background sentences for example and I think that TL DRS could potentially be more informative other places we might see this would be paper recommendation feeds for example but I I would like to see maybe as future work more evaluations and see like what exactly is the best use of fuel ers but I think that the there's a lot of possibilities for how we might be able to use this work in the real world okay great those are great there's a great example is I thing so I want to move on now into since we already heard some practical as the side of it which is great you know they were pressuring me by the way class I question is always a question that comes up I was going to ask it it's also about the evaluation metrics which is something that always comes up as well evaluation metrics and you know recently at least in a healthy community we have been discussing about evaluation metrics a lot and I saw a lot of researchers coming up with new ideas and how to create better evaluation metrics for different LP gas so what would you say is the as I remember like in the paper you mentioned that so we use rope or evaluation metric use common commonly in text generation right so text summarization so this evaluation metric so do you see any way by the way in how you can improve at least on the evaluation side of things how would you be able to improve it so so any any comments on that any insights on that yeah I mean we so so rich is like the standard for summarization for automatic evaluation as it is right now but I think that there's there are multiple papers out there talking about some of the issues with revision certainly there's but there's a lot of room for improvement in terms of automatic evaluation and I think we've seen some recent work of researchers looking at this issue because yeah definitely there's a lot of room for future work and about automatic evaluation of summarization yeah because I also noticed that so you you you actually in the testing phase we haven't spoken up with that but in the you know when you collect your desk test dataset you actually had like multiple goal right so so we'll go examples rather just using one was that was that because of the evaluation metric that was used why was that kind of reasoning behind that using you know besides the variability part I said the variability part was actually a challenging thing in this paper I guess and it's work so what was the what was the reasoning behind it particularly how it's connected with the evaluation no yeah so part of the motivation behind having multiple targets in the test set was that when when there's there's always going to be some variability and human summaries you know the TLDR that I might write about paper might look different from the TLDR that you write but could that it could both be perfectly valid to yours so our thought process was that whenever we're thinking about automatically generating till the RS we don't necessarily care if that the model generates my TL DR versus yours we care that it generates something that is feasible that humans might write it right so our thought process was that okay so brugge tries to match the it measures the overlap essentially between the predicted summary and the reference summary but we don't necessarily want to penalize the model for generating something that's more similar to my Tod or as opposed to yours okay so can you can you provide us like an overview also on the very short overview on the different input configurations that were used for this particular task so I know you used in some cases used actually the abstract to generate you know that information to generate the TLDR or in some cases he also combined like full body text and also in some cases you can kind of combine abstract introduction as well as the conclusion so I've done really similar research back in the days I was doing like keyword extraction and I had a very very very similar kind of hypothesis that you know you can select information most of the important information of whatever was written it's kind of either in the title or in the introduction right at least at the higher level that information is actually contained here so what were the what are the major findings here in terms of the different input configurations that were used right the different combinations at the abstract so what was the kind of what would you say overall is that kind of a major in finding um so for using the abstract only the using the abstract only I think in part was motivated from the application point of view and that production level systems you don't always have access to the full text so we wanted to see if we could generate reasonable TLDR from just the app tract but I that if I asked you to write a tldr from a paper that you would just read the abstract and then write killed ER you'd probably read the whole paper right so we were looking into what parts of the paper might be most useful for generating a tldr and you know this is in part because of just practical limitations part has a Mac sequence life that doesn't fit the full context of the paper for example and so looking at previous work that has shown that the abstract and current interim conclusion contained the most salient parts of the paper as well as some of our own analysis where we found that the the Oracle scores jumped more from abstract to abstract into conclusion than it did from abstract into a conclusion to full text we thought that this would be a reasonable input space to try out okay yeah so just of that topic as well I also want to ask like in terms of the input configurations because I think that's that's the word that kind of the research could also be you know people can actually explore it a little bit further as well I'm curious like do you think there could be an opportunity here to actually explore so explore for instance the use of knowledge bases and different kind of external resources to be able to come create better TLDR so to speak so to speak I think that's something that's possible and worth exploring like using external resources and what are your thoughts about this yeah absolutely I mean I think that there there's a number of different input sources you might be able to try to help generate TLDR as well including an external sources potentially right because I also you say mentioned also the importance of correctness right so when you are generating something at the ldr you want to make sure it's high quality but you also want to make sure it's this factual it's actually correct and accurate so I think kind of the use of knowledge base is to actually have like an automated maybe fact-checking process be something like maybe worth exploring just an idea there but I'm just curious like in terms of like future work is there anything that you're kind of exploring any kind of questions extended questions to this research that you know could be interesting and you can comment on yeah so I mean I think that there's a lot of directions in terms of future work a really big one I think is so that so the data set that we introduce is English language papers in the computer science domain so I would love to see future work expanding to other languages and other domains I and I also think that like in our in our in our paper we make some assumptions about the background knowledge of the reader so it'd be really cool to see future work maybe trying to explicitly model for the background knowledge of the reader the TLDR you might give to your colleagues might be a little bit different than the tea of the art you give to say a journalist who's writing a piece about your work so that's a really cool potential future direction maybe using our annotation process to for example use Twitter as the input as opposed to peer review comments and maybe constructing a more informal version of the data set yeah there's there's a lot of potential future there work that would be really cool to see definitely I think in terms of application as you mention as well doing it in different languages and see what comes out of this if there is any kind of new insights we can pick up as well that we can learn better you know improve generation and things we can learn in different languages as well we can kind of add also to our own language so I think this is a really interesting ways to keep exploring and the same research so that's why I think it was exciting that's why I kind of wanted to get you here just to kind of ask you about it and then just to get some some insights from you so that's great thanks so I want to ask us well so of course on the in the conversation of theall generation we kind of understood what was the motivation at the beginning when you explained you know just trying to keep up with the fast pace of amel and publications just left right and center a lot of publications everyday it's really hard to keep we even have a newsletter to actually help with this like to keep people up to date with this content it's really hard so I'm curious um in terms of you know you being a researcher now you know investing a lot of time in doing a really really great research how do you how do you actually I want to know like how do you actually keep up to date with everything happening around you in terms of ml and it'll be research how do you actually do that yeah AHA men keeping up to date with literature is one of the struggles of painting a researcher right yeah so small plug personally I really like semantic scholars research recommendations need if you go on semantics color you can create a recommendation seat of papers and you can base it on certain topics so I have one for summarization for example this is one way I like to keep up with research of course Twitter yeah I guess everyone here is using Twitter it's somewhere the other I'm guessing that yeah but so much this color is pretty interesting I've started to experiment it with an experiment with it by generating my own like 10050 feeds I think that's a really really good idea because obviously this thing is coming at there at the such high volume so you need something actually create this kind of person is a personalized list for you so I think that's a I said that's a really really and great tool to use so you know for those of you listening here I think that's a really great tool to kind of I'm trying for this one so yeah I mean it's all related right it's all ready to actually the work that you're doing in how to better improve surfacing the information that matters to people just to give people and that value of looking at something pretty quickly and deciding for themselves whether they want to continue reading further so I think that's really really valuable there um so I guess that those were kind of the questions I wanted to ask you that's going to be on the 30 minutes that we I promise that maybe we keep it to 30 minutes or 45 I want to take too much of your time here and you you've provided a lot of really great answers here I think it's a lot it's a lot of value we get from you here having you as one of the quadrants of this paper and just to get your insights as well and how curious you are about this research and the areas you want to kind of further explore so really really thank you very much for doing this with us here so I just want one thing they want to say also if you have any any of course anything that you want to mention any kind of last words any you want to plug anything are you writing about any particular research so anything you want to share anything that's useful for us as well yeah go follow me on Twitter I don't currently have a blog or anything like that but I imagine if I made one I would announce it on Twitter so I knew your accounts or whatever when I polish the recording I'm going to actually provide information for everyone so that they know where to find you okay so so you don't have any website or anything like that that we could you know I I do have a website it's Isabel controller comm um yeah in terms of like day to day things Twitter but I'll also keep keep my website up to date with publications and stuff okay they'll be really helpful thank you for that so I guess so bit I mean thank you so much for doing this and for taking the time to speak with us we really appreciate it as a community that you were willing to do this with us today so we're gonna have the recording and that's gonna be great because let us see people now can always refer to people if they're interested in this research they can always go to recording and check a little bit more and what we were discussing some of the questions so thank you you said well for doing this again I think we're gonna let you go now thanks a lot for doing this we hope that you know we can keep connected and maybe you know keep putting more work and eventually in the future maybe if this would happen again we really would appreciate it so thanks Isabel and have a good day