"What Can We Do to Improve Peer Review in NLP?" 👀

Key Takeaways

[Music] hello to everyone miss coffee bean here finally letizia has found some time to animate me and give me my voice back i missed being out here with you guys today we are going to talk about peer review in nlp something that leticia has been busy with during the last few days and in this video we will be presenting this awesome paper by anna rogers and isabel augenstein it is a short paper accepted at mnlp findings this year it discusses how to improve peer review a topic that could change the world well could peer review really change the world yeah why not think about one of the main chains of ideas research with new ideas coming not only from academia but also from industry should be spread to others one way to do this is through publications what ideas are valuable and impactful enough to be published is determined by peer review where other scientists get to say how much they like the research and how much not after publications the ideas can be taken on by others for improvement for further spreading or integration into super cool machine learning applications right but how to do the application part properly if the first part of the chain is flawed because surprise surprise peer review is broken if we want to know how to fix it we also want to know how it is broken so it is time to look at this paper even though the paper is about the natural language processing field we think that the general message can be useful for any kind of peer review so let's see the paper really does not mess around and starts abruptly with harsh claims like peer review does not guarantee quality control ouch but no matter how much this sentence hurts we cannot but agree to what they argue peer review is meant to guarantee that what is written in the paper is correct and high quality but this is an impossible task because the papers are not reproducible especially when thinking about the poor reviewers with limited time and unpaid review work then the others say that peer review fails to detect impactful papers again very harsh but quite true important for paper impact is often not only the science itself but other factors like the topic the promotion on for example the social media or how easy it is to take an idea and build on it further look at for example bird or transformers okay so surprise surprise peer review is hard and ill defined so the authors ask what could peer review realistically do well it could filter out obvious flaws and turn the spotlight on to quality right but the problem is that we are not doing that instead we are selecting or trying to select the best x percent of the papers and accept them what could possibly go wrong here by ah yes we do not know where and how to draw the line the idea that miss coffee being appreciated the most out of this paper is represented in this figure reviewing is like comparing apples with oranges different papers have different strengths one has better methodology the other one has better evaluation or should we select the idea but not so much its implementation or what should we focus actually on i don't know how to judge this i'm not gonna actually my special so while miss coffee bean is having a mental breakdown i can tell you that the other thing that i liked most about this paper is the idea that reviewers choose different ways of coping with this impossible task that is required from them so miss coffee bean now that you recovered from your breakdown what are the coping mechanisms ah writing style it is hard to resist the temptation to think that if there are language errors in the paper the science is bad too but this thinking is so wrong non-native english speakers can produce awesome science too well the next way to cope with the impossible reviewing task is to look at only one table in the paper you know the one containing the result if the proposed contribution is not the best there we reject we want only state of the art right you know but this ignores many things like efficiency interpretability all the other goodies beyond a high score in the evaluation metrics niche topics like you know everything that is not transformer or is not i cite the paper scientifically sexy is so hard to publish especially in natural language processing it is easy to publish work on english language but it is hard on others as if findings on other languages are not as generalizable as the findings on english to this wall of shame the authors further add the problem widespread in the whole machine learning area like already famous work where researchers publish already on archive so the anonymous peer review is not so anonymous in the end also big labs which invest a lot of work into public relations are famous and they get even more famous in my observation the big labs are even more recognizable in review just by you know compute resources or even plotting styles so let's move on to the next coping mechanism we sometimes tend to shame two simple solutions because simple means it is easier right nope it's not over complication is easier than simplicity another way of coping is through spotlighting mainstream approaches like deep learning but no deep learning is not the answer to everything and other methods have their merit too and especially in the big data requirements of deep learning it is quite unintuitive why we do not appreciate resource papers so much so okay we're almost at the end of the wall of shame novel approaches are too often rejected this sounds absurd yes and the authors explain it through the human way of saying no to new stuff because it is hard to comprehend especially at first and the last and i think by far the worst entry of the wall of shame are substitute questions that reviewers tend to ask in the impossible situations of assessing quality if i did this study would i have made the same choices and in our opinion this is the worst coping mechanism because it is the most subjective one and has to do a lot with the different backgrounds and personal ways of thinking of reviewers i mean the other coping mechanisms are at least expected and easy to counter but this last point about the substitute questions is as varied as there are reviewers for example if you write a paper and test your model on data set x and data set y the reviewer says or asks but why did you not test on data set z the reviewer might actually mean if i did this study i would have tested on z and the problem with this question is that it is hard to say if the reason for having data set z is really scientifical or just you know personal so anyway while looking at this wall of shame one might come up with the idea just to abolish peer review but we remind about what we discussed at the beginning of this video where we need to have a filter for publications for non-experts so they know what's quality and what's not so what to do the authors of the paper propose that reviewer work should be more valued and the working hours taken into account by employers review work should really not be done by overworked people in their spare time and then they propose some ways to reduce the uncertainty that comes with the poorer task definition of reviewing for example more organization energy should be invested into reviewer matching for assigning competent reviewers for each research topic then the review task should be more fine-grained to capture the diversity of papers and also the review forms should be able to capture this announcing the conference or journal priorities before submission and reviewers giving not an overall score but specific scores that are weighted differently depending on these editorial priorities should also improve the definition of the task of peer review and the reviewers faced with a well-defined task would be then relying less on coping mechanisms okay so when do we do this many others had the ideas presented in this paper and what i really appreciate most is that the authors managed to remind here by everybody again about the peer review problems so maybe this wake up call will finally have some effect miss coffee bean wanted to spread the word further with this video if you want to help too share this video with anybody interested or share the paper you can find the link to the paper in the description below see you next time okay bye