NeurIPS 2020 Changes to Paper Submission Process

Key Takeaways

hi there so um I just wanted to give the few quick thoughts about the changes to the Europe's submission process this year as opposed to last year they've announced this on the website on Twitter with the video and so on and I thought I might share some thoughts on that and maybe some of you haven't heard yet in case you're planning to submit or thinking about it so desk rejections it ACS area chairs have the ability to desk reject papers that they feel strongly are not going to be passable to the reviewers they did an experiment last year where the ACS were simply supposed to mark submissions that they would desk reject and it turned out ACS aren't very good at estimating which submissions are going to be rejected by the reviewers that might be because there wasn't really anything at stake because it was just kind of uh let's see how this works but it is definitely a move to reduce the number of submissions because the field is exploding and we lack reviewing power reviewing people so this is a move to reduce the number of people that have to review something because there will be fewer fewer papers I don't know if this increases the quality overall if your paper gets desk rejected there's usually like some obvious reason for it why an AC decided it's not worth it they probably haven't read it in depth but there there might be some kind of overall structural issue that or like the introduction has many typos or you know that like look for the obvious things even though your work might be good right right second all authors of a paper have to be able to review if asked to do so and again this is this kind of reviewing crisis I have mixed feelings about this like I really think this is a move in the wrong direction it will increase the number of authors because a lot of people have been kind of freeriding in that they're submitting papers but they aren't reviewing other papers even though they would be competent researchers simply because reviewing doesn't get you anything so there's no incentive to do reviews maybe you can say you're a reviewer but then there's every incentive to do bad reviews like two-line reviews where the first line says you should have compared to my paper reject like [ __ ] you with your reviewer like this in any case like a lot of times and this hits for example like universities where you maybe work with a master student and the master student does some of the pre-processing of the data and right and they don't really have a clue about the machine learning but they still contribute it so why shouldn't they be an author on the paper they might even have been have written that section about the data pre-processing and now they're asked to review entire papers about topics where they're not really familiar with or you have some outside collaborators or you know that there's so many so many things wrong I think this attracts the wrong kind of people and by forcing people to do it you encourage even more like all these reviewers that would not have reviewed what will happen is they will give shitty reviews and you will have even worse quality of reviews as a result I think this is the wrong move to reduce the number of load per reviewer I'd rather see abolish peer review completely in computer science in machine learning at least that's my opinion but that might be a video for another time I have plans how to replace it another time resubmissions have to be clearly marked so if your paper is a resubmission of like if you had already submitted it in the last 12 months it's been rejected you have to say it is a resubmission and changes you made to the paper again with with a with a peer-review process that actually works this would make a lot of sense you can say well it got rejected last time and here is how I corrected for what the reviewers criticized but with review quality right now I mean most of the papers what are they gonna say it got rejected for nefarious reasons because the reviewer had a bowel movement that morning and I didn't really change much so you encourage people to kind of blow out of proportion the changes they're made and put a lot of additional unnecessary work on to papers that would actually be already fine right so this all of these things that they'll just they are they're forcing people to do things and then the incentives of what we want aren't aligned with what we give right so what you'll end up with is lower quality ÿû reviews and lower quality work so the next two points are of a different nature the first one though I thought that would probably I mean even if the AC's aren't perfect you know that that's a bit I like that the fourth point and the fifth point are a bit different the fourth point is there is a new section in CMT apparently where you have to describe the broader societal impact and ethics around your work like how will your work influence society what are positives and negatives ethical outcomes how can it be used and this is targeted towards things like let's say facial recognition if you develop a new facial recognition algorithm you may be able to argue well this could be better used to identify you know um victims in a big crowd you know there's a mass riot or something and then you don't know who was there is my relative one of the people in the mass right that gets stomped on or you can also say this potentially helps a dictatorial state to govern their people because they cannot recognize everyone for most papers it will be a bit shaky like if your third order optimization algorithm that she used a slightly better convergence rate I'm not sure what's here but what what I feel is that this this is it's dumb in a way because this just means more work right basically now you have to demonstrate and yeah it says you just should discuss positive and negative aspects but in essence everyone will be demonstrating virtue signaling how good their work will be for society and what good can be done and maybe a bit of bad but that can be mitigated right and and it just pushes it into a more PR world so it goes from the science world into a more PR world it means extra work and who are the people that can afford to do extra work it's mostly the big companies right they can just put an additional team member on that maybe even do additional experiments to show the societal impact of the work and who will lose out or probably small universities independent researchers and so on that don't have that that capacity that simply do their research right because it's an interesting research question and for almost every single thing in the world that has an application it will have good and bad applications so yeah mixed feelings so fifth is you are now supposed if your paper gets accepted to make a video about it and upload the poster basically link to the poster that you would use and also link to slides that you would give their talk with this is to make it more accessible to people people that are not at the conference which again I have mixed feelings about again it pushes it into this more PR round right talks are already live-streamed most of them are for most of the large conferences and I feel it just gets people one step more away from the actual paper like it's very so it allows people to grandstand and in PR up even more of their work because even people who don't attend the conference now they're not going to read the paper just gonna watch the video right and in the video you can always leave away those you know things that you would have to like that a reviewer makes you put in the paper right and in the video you can over voice things camera ready no one reviews the video you can say whatever you want so it's it's just where before if you didn't attend the conference I think many people actually did read the paper watched talks where people could ask questions and now it's it's just one more PR thing and again who has time energy and money to really invest a lot into this it's mainly large companies right if you're small and you're time-bound and so on you might not have equipment or time to do that I am NOT for hire to do your videos just saying I don't have time to make these videos really as you can see in the stellar quality I think there's a bright glare right here so that was it for my opinions on this and I wish you nice day bye