arXiv Publication Patterns

Key Takeaways

[Music] welcome to data skeptic machine intelligence our podcast series exploring contemporary topics in artificial general intelligence and large language models today on the show we're going to talk about one of my favorite websites it's archive.org arxiv.org as we'll be somewhat obvious to people who read our show notes I would say 90% probably maybe 95% of gu are people who've previously published on the archive and we're going to talk about a paper they published there we actually crawl the archive over here at data skeptic HQ we compute Bert vectors of the titles and abstracts and that's the main input to the pipeline we use for finding guests on the show much like our guest today who also did some research on the archive in this case a deep dive into questions related to large language models and how they're affecting research safe to assume there's an increase in Publications on this topic but how much much who's publishing and what kind of crossovers exist all that and more coming right up my name is Rajiv MOA I also go by Raj mostley I'm a PhD student in computer science at Cornell Tech Cornell University has a basically Tech oriented campus um for graduate students in in New York city so I'm lucky to be in New York City I'm specifically you know still figuring things out to some extent but have been interested in NLP for the last few years and and more specific spefically sort of the social implications of machine learning and NLP systems NLP natural language processing there's a lot of work to develop new models but how will these models actually impact Society adjacent um Society facing applications for example health law or these sort of adjacent to real world uh applications I'm I'm more interested in that than sort of like doing very core sort of methodological research I'm more interested in thinking about what are the impacts of models and how do we make sure they're more likely to to do good than than bad well it's either incredibly exciting or incredibly terrifying time to be studying NLP uh maybe a little bit of both what's it like being kneep in that research as the field evolves quickly yeah so I mean I would say that I'm to some extent an outsider to NLP in this paper for example we sort of you know have some foot in what's going on with NLP but we're also trying to sort of study it from an outside perspective and see what's changing about NLP so that's question about like you know how are things evolving and what's changing and what's it like is exactly what we tried to get at I mean sort of my on the grounds perspective is you know again as someone who does do a little bit of NLP but also is interested in sort of social implications I think you know that field is also moving very rapidly you know figuring out how to steer these models in a good direction also figuring out like sort of what are we actually even concerned about like Are we more concerned about short-term harms or long-term risks is another thing to think about I mean overall I feel like there is a lot that's going on it's very hard to keep up with and and that can sometimes be overwhelming um but I also think it's it's interesting and exciting I think it's fair to say we're in the early days of large language models being used in society in different products or in decision-making processes do you feel that that's kicked off or are we about to see a wave of these sorts of implications yeah like you're saying we're still in in early stages I mean there are some applications like for example if if you're a software engineer like I think a lot of software Engineers nowadays like actually use sort of co-generation assistant and most notably github's co-pilot like I use co-pilot that's that's probably the primary way I interact with language models on a daily basis is co-pilot um so so there are these some niches where you know they already have pretty high adoption you know working them into like customer service or other sort of very tangible products you know is will probably impact much a much larger fraction of the population and you know we haven't seen as much of that yet but um we're probably going to be looking at a lot more AI generated text um you know in the next few years well I know in the paper in the research you've been doing you're able to get a sense of publication patterns how people are at least writing about their research using large language models how do you come by something like that what's an underlying data set that can help answer questions so we use um the archive so that's a website where um it's not peer reviewed it's a pre-print server where a lot of researchers especally in computer science post their work archive originally had high adoption in the 2000s uh among like other communities like physics but yeah in the last 10 years I think most computer science research gets published in conferences eventually but also gets posted to Archive so that was a pretty good data source because it's it's all freely accessible there's very sort of uh new work that gets posted to Archive because like I was saying it's a pre-print server before before people actually get their their work peer reviewed um so there's a lot of stuff there it's a great way to sort of capture like an on the grounds look at at things that are happening you know now like every day there are many new llm related submissions to the archive there are some things that are missed I think from archive but also like looking at papers more broadly like some companies don't really publish their work as much um and so we wouldn't be able to see you know their internal research internal research obviously but I think you know to get some summary of what's going on archive is a very good place to start and that's that's what we used and is there any metric you have in mind by which you can scope the size of this how how big is the data it really comes down to how you slice it to get a specific number um but you know so what we looked at we started with everything that was posted to Archive within a computer science subcategory within the last five years since 2018 which I think you know everyone would agree that models before 2018 were probably not really large language models so we started we started the sample in 2018 took everything that had been submitted to to Archive in a computer science category so that that's 400,000 papers and then we asked out those 100,000 papers how many of them have sort of any keyword in their title or abstract which is related to language models and that's sort of how we we defined a a set of papers to analyze and that turned up 177,000 papers so that's quite a bit and you know not every single Paper in that set would you know have unanimous agreement that it's a large language model paper but it gives some sense of like the scale of of you know how much research there is uh in this space is like you know on the order of 10,000 papers which just out of curiosity you went with a keyword search I consider that kind of an old school technique you could also have just submitted each of those abstracts to a large language model and asked it do you find this abstract to be about a large language model was there any you know kind of design choices or things like that you had to make along the way yeah totally that would definitely be a possibility yeah I think with the key research you know within this field of like science of science is the term you know to science of science being like you're doing science on the Enterprise of science as a whole so so that I would say that's you know the broader subfield which this work could be contextualized within and I think within that space it's important to do your work in sort of an interpretable and in a way that's easily reproducible and um has sort of a clear understanding of what was done and so I think a keyword search like made sense in that broader context of like science of Science and then also I think you know have start with a very broad set I think if you like ask an llm is this paper llm related that that that might give you good accuracy but like we actually weren't too concerned about false positives with this keyword search and if you have a sufficiently broad set of keywords you're you're probably going to capture most things which could plausibly fall in into llm related so yeah the reason why we we wanted a very broads set was because we could we wanted to analyze sort of changes and see how this research community research base has has evolved uh in the last few years we felt that like having some papers which were not totally LM related like that would sort of naturally Fallout out of our analysis overall we wanted to prioritize like recall over making sure every single last paper in the set was like definitely LM related well I think it's a safe bet to assume it's monotonically increasing in volume but do you have any high level observations about the trends in publication yeah so when we first started the work we just wanted to look at you know how fast is this field exploding and so yeah we I think we have a supplementary figure which which mentioned this and basically it seems like over a 10% of all computer science research seems to be somehow llm related nowadays basically you know several factors higher than it was a few years ago even so clearly a lot has changed just in this last year yeah so we started there by just looking at the quantity of papers but we actually realized that you know everyone kind of knows that things are increasing quickly so most of our analysis actually focused not just on quantity of the increase but more trying to describe how things have changed who has been responsible for the increase what institutions are are the biggest players Etc what are some of the the broad other secondary analysis you want to do you know the high level counts important but as you were mentioning who is publishing more uh could you perhaps just outline some of the overall slicing and dicing you set out to do I think the most surprising thing for us was the extent to which you know this rise in llm research centered around applications and societal impacts so previously natural language processing researchers more were more concerned with sort of specific tasks within NLP so for example people would would care about question answering so if you ask Siri a question about like a fact can it pull up the relevant Wikipedia article and serve you an answer and we it get it right so so sort of very narrow tasks like that and now people are using llms obviously for sort of all sorts of things from sort of code generation to healthcare and and sort of health information and you know solving medical challenge problems or analyzing legal documents um recommender systems just really all sorts of things and for example we saw that so papers on archive can designate a category or within computer science that they're related to so we found that the the category within archive that was actually Rising the fastest by far was the subcategory of computers and Society um which is sort of deals with all sorts of societal implications of of technology and that that was the one which which had the the greatest rise in in llm research so I think that was very surprising sort of topically how much the quantitative data confirmed our intuition that llms really are having quite a big impact on society and will continue to another application that I should have mentioned earlier which I think is maybe one of the biggest ones is is education so not just you know how are LM going to be useful in education but there are all sorts of people writing about will llms actually be net beneficial or harmful in education um how do we like rethink examinations or like application processes for example in light of the fact that llms are are are so performant nowadays um so all sorts of this stuff was very interesting to to see pop out in the data so yeah that that's that's one sort of a topic based analysis another analysis is an author-based analysis for example what affiliations are are rising the most and what affiliations are accounting for the most llm research and it's important to note that you know this affiliation analysis for example this is about the research landscape it may not necessarily reflect you know the fact that everyone uses chat gbt and that's that's from open AI but open AI doesn't publish that much which is one thing we found in our in our work and I think people kind of broadly know we actually found that yeah so Microsoft and Google are the you know top two industry producers of of llm research just by the count of the number of papers they they've written about llms and then after Microsoft and Google and a few other large industry players actually the long taale um and the vast majority of of work um is actually from you know universities and and academic institutions a lot of expected ones like CMU Sanford a lot of top Chinese universities too like squa a ton of a ton of academic players that are having a huge impact in the research landscape even though we may not hear as much about their work as we do about Google and openai and meta and stuff sort of in the public discourse well Google and open AI are certainly pouring lots of money and lots of talent into these areas when you look more to the academic institutions does it require the same sort of uh I don't know Firepower is the right word is it large institutions that are publishing a lot or is there room for smaller groups as well so so we actually saw that if you look at academic institutions versus industry institutions the total percentage of papers is actually higher in Academia this year compared to before this year which may seem counterintuitive because you would think that as these models are getting bigger and bigger sort of you need more and more compute to actually use them you know compute is generally centered in these very rich well-funded profitable industry institutions as you're pointing out so so how are academics actually publishing more a larger fraction of of these papers you know the main difference between academic work and and Industry work is that Google Microsoft uh open AI they they they sort of have the Firepower to train their own models and fine-tune their own models and you know do all sorts of things in the training process less of that work that sort of methods training work is happening in in Academia and more academics seem to be using sort of API access of model so people a lot of academics use the open AI API to access gp4 or maybe we'll use like a small version of llama Etc so they're using either open source models or or sort of API accessive models to study them or apply them to societal applications instead of sort of training training their own models because like you're saying you know most of that can't really happen in Academia so I view this as you know interesting and also researchers from Academia like like us are able to actually study the models that are being widely used and and you know people can audit these models for example audit them for bias and harms to get a better understanding of are they going to be beneficial or harmful that said I think it's you know you can only do so much if you don't actually have the underlying weights of the model so no one has GPT 4's weights or even knows how many parameters went into GPT 4 and that limits the number of things you can study about it that is a you know a rift between industry and Academia that will only probably continue to grow as details about the models become you know secretive and they get bigger Etc and and so we study collaboration between different institutions in our data set we we want to see you know are there some types of Institutions that are collaborating a lot and also more interestingly is is there notable the lack of collaboration between some types of Institutions because in so far as there's you know a notable lack of collaboration there might there might be competition um and competition could have other Downstream impacts we Define a collaboration as a paper which is written by more than one more than one institution so for example you know Microsoft or Google write a lot of papers with universities like Stanford CMU Etc so there are a lot of uh industry Academia collaborations which kind of makes sense you know a lot of PhD students intern at companies are over the summer for example and they write papers together but we saw uh you know a very notable Darth of collaboration between industry institutions Microsoft and Google almost never collaborate um pairs of Industry institutions very very rarely collaborate and you know this isn't crazy surprising companies care about protecting their their IP Etc but it is interesting to see that Trend confirmed in the data and and also potentially concerning because we've already sort of seen what happens uh when that's the case so for example after chat gbt came out we saw Google and Microsoft you know rush to release their own kind of chat style models in uh Bard and and B chat respectively these models weren't as performant and clean to use as chat gbt and and more importantly they maybe had like actual ethical concerns and and led people to do things which were unsafe that may have been the result of this pressure for these companies to put out models earlier than they actually should have been released you know as a result of competition so that's one thing we've already seen and we also saw maybe also not too surprising but we also saw a lack of collaboration between us and Chinese institutions so us and China both industry and and universities put out the the most llm papers and and the largest models but they very rarely collaborate uh in our in our data as well um and we think this trend holds in general there could also be important sort of concerning implications of of that lack of of collaboration um obviously our paper doesn't speculate as to what those impacts would actually be but yeah comp competition could basically could pose harms well as it stands today do you see any major differences between the types of topics that industry is looking at and Publishing about versus what Academia publishes about yeah we we definitely see differences so I alluded to them a little bit but yet to go into more detail on them so so industry is doing seem to be seems to be doing more sort of methods work around training so for example a lot of work in industry is around like efficiency and performance their techniques like pruning which is removing some of the weights of the model which you can do without really impacting performance there's a technique called quantization where you use a lower bit representation of Weights um in order to save memory for the model there's a very big technique called knowledge distillation where you can basically take you can benefit from the accuracy of a large model and actually use it to make a small model better so industry is working on all sorts of these sort of efficiency oriented contributions to make models more Deployable and usable for their products because you know they care about the cost of deploying a model and uh as well as these sort of more traditional core NLP tasks potentially like search and retrieving information and answering questions that's where we see more of an industry role and then in contrast academics uh work a lot on like applications uh so Healthcare is a big one there's a growing emphasis on using language models for legal document processing and legal analysis I I don't have you know great understanding of exactly what's going on in this space but we see that you know legal related work is popping out a lot um in the data that's some sense of of the differences between industry and academic work you'd mentioned one area I'd love to zoom in on a little bit and that's llms and education uh I personally am very optimistic about the way these can be addative to uh a students experience and help them learn faster and more efficiently and things like that but even though I have that expectation it should be science that tells me if I'm right or not yet we've had these for such a short period of time I'm not convinced we have enough like longitudinal data to answer any questions do you have a sense of what the research gets into and what we can say at this point in time about llms in education honestly it's not my area of expertise either I did see some of these papers pop out in our data as as you know education being one of the fastest growing topics people are writing about but yeah I think people make Arguments for you know both benefits and harms I I don't have a great sense of what the consensus is or if there is any and I think you're right that they're going to need a wait to see how it shakes out I mean one thing that I've thought about a little bit and don't really have a great answer to Is like how LMS or writing AIDS might change like college applications because application essays have historically been such a big part of how a lot of universities in the US at least make their decisions and if you ask chat gbt to write like an essay which you know describes you know a pretty personal aspect of your life you know elaborate on it in a personal statement length essay like I feel like it does a pretty good job I don't know I'm I'm concerned to some extent or are just kind of curious about how college applications will shake out I don't know if you've read anything on that I haven't but I I agree with you I think it's a a point of inflection in how that's all going to work yeah so similar question when it comes into societal impacts I think it's it's in many ways too soon to say what some of them are uh I'm I'm sure there's good room for speculation by thought leaders but do you find that we're living in an empirical world when it comes to those sorts of analyses yeah I mean I do think that a lot of the works so far like an education for example or law a lot of it is more in the kind kind of trying to guess regime um or like listing some OPP opportunities and challenges you're right that we'll need like sort of more time before empirical things can actually happen I think with with some domains like again I keep going back to code generation because I feel like it's one where llms are really already having impact um so in that regime for example there is this one interesting human computer interaction paper that compares like software Engineers ability to solve coding tasks like real world examples of things they might do in their job with and without an LM assistant or co-pilot specifically um and they find that for some tasks there's actually a solid benefit in the amount of time it takes to complete it so yeah there are some domains where some empirical work has happened but overall we're going to need a bit more time before we get a clear sense of what the quantitative possibilities are here and I'm wondering if you could share a few details on I guess the engineering challenges in a project like like this I'm sure it's not trivial to manage this much data and do some analysis what was the work like one one of the challenges here was we have 17,000 papers and we want to organize them in such a way that we can make observations and trends that are digestible and the sort of core of that was assigning papers into clusters and then labeling those clusters with topic names both of those steps are not trivial there's a lot of work on clustering you know data points in general in machine learning in data science so we were able to use a lot of tools available for that but even like how do you convert a paper into like a mathematical representation is is difficult and and so we used um you know another sort of advance from NLP we use embedding models which basically can take in any arbitrary string of text usually up to some limit of words and map that to a a mathematical Vector representation so we did that for the paper abstracts only um and obviously abstracts are don't tell you the whole story of what a paper is about but they give you some solid sense so we took each paper got a representation of its abstract and then clustered it using standard methods and then now you have a bunch of like clusters but what are those clusters actually about and how do you label them succinctly that's another hard problem and and we actually funnily enough used gp4 to help with the labeling so we took these clusters each one had about 500 2,000 papers we took a sample of like 25 of them threw their titles and abstracts into gbd4 and said what's like a su cluster name for this set of papers gb4 unfortunately didn't do you know that great but it gave sort of a candidate start and then you know was a lot of manual work to actually go through and ask is this reasonable do I need to like refine this and most of them I needed to refine um but yeah so sort of that work of like actually taking 177,000 papers and designating them into a more adustable number of topics so we ended up choosing 40 topics that was one of the more tedious tasks of this project but then once we had those topics you know a lot of the analysis was easier to do because then we could go and say okay I know this paper is by an industry author I know this paper I know these other papers by academic authors um how do these two populations um industry and academic authors differ in their topic count for example and do you have any thoughts on maybe where this goes next you have some really novel conclusions and results but it seems like a endless opportunity to go deeper into the archive data and get other insights if you had infinite time what would be some of your Pursuits yeah I think sort of a sub question with with all of this changes and advancements in in AI is like what will the impacts be that's that's been something we we've talked about there are two different camps kind of that would tell you different things um and also people study the implications differently so yeah I guess specifically I'm interested in this question about these two communities uh the AI ethics community and the AI safety Community which you know if you're not sort of well versed with this space they those may seem to to sound pretty similar ethics and safety but within this community those two words actually mean something pretty different so AI ethics tends to focus more on near-term harms things like algorithmic bias so uh will language model generate text which is biased against certain groups will it you know stereotype minorities for example in its outputs so those sorts of questions are are you know generally more concerned with the AI ethics community and then the AI safety Community is has been in the popular press and news a lot recently and and they're more concerned about long-term like harms and and existential risks potentially from AI so in so far as AGI is a possibility should should we be concerned about Ai and destroying humans and if so what can we do to prevent that so these both seem like very important problems but these two Fields don't overlap that much in in terms of research communities and it'd be interesting to get more sense as to why that is and sort of assess first you know concretely what are the different problems these two communities work on and why do they you know value these problems over the other communities problems um so why is it that some people think that sort of near-term harms are critical to are more critical to focus on and why is it that the safety Community for example thinks that existential risk from AI is a real concern that more many more people should be working on we could make some Headway on this question from the data we already have of papers because I think you know some some of this conversation happens in in the research space we we would have to look further than just papers and and look at other forms of documentation and other statements and petitions and all sorts of stuff to get a better understanding of really the Divergence between these two communities well if uh you had told me that 10 years ago I might say well you know the Chinese language is so fundamentally different the statistical models they're going to have the structure of grammar everything is you know they're going to need totally different algorithms and tools is that still today yeah that that's that's a good point just the underlying fact that you know English and Chinese are very different um is probably relevant here there are a lot of NLP papers coming out of China that are being published in top venues that focus on English as as the as the language so I don't think the language difference is explains the whole story I think there is potential for collaboration in that people are working on on similar things in both the US and China but there's still competition due to other reasons like more soci technical reasons and I'm curious what's next for you in your research yeah so I'm still relatively early on in my PhD um and I'm very much still figure figuring out my my my research interests I find that this style of work of you know trying to analyze science from sort of a higher view like sort of meta studies of science I find this to be interesting and I think it reveals a lot of things which people take for granted and may not think about as much as they should be thinking about so I I like the style of work that said I also have interest in like actually using language models improving societal outcomes so I'm interested in the potential for large language models in health app applications specifically applications that might improve Health Equity um so people write a lot about how LMS could be harmful for equity and and bias and fairness but I think there's there's a lot of room for these models to democratize access to healthcare for example which is traditionally very siloed and there's a huge power imbalance between doctor and patient and I think llms have the potential to increase access to information in a way that could actually improve Equity so that's something I I would maybe like to explore more from in a more experimental setting yeah I'm also interested in sort of the core methods Behind These models to some extent for example the reinforcement learning for human feedback space is is very interesting and and underexplored in some ways um and so I I have some ideas there too but generally just just trying to figure things out in this very rapidly changing space well lots of good directions and opportunities for sure and is there anywhere listeners can follow you online yeah I'm on Twitter may maybe my handle can be posted in the description but yeah it should be easy to find yeah we'll have a link in the show notes for people to follow up Raj thank you so much for taking the time to come on and share your work yeah thanks so much Kyle it's really nice to be on the [Music] podcast