🔥 ChatGPT-4 beats Human Annotators

Key Takeaways

gpt4 powered charge GPT has successfully beaten human beings in text annotation tasks it's in zero short learning which means there is nothing else been given to gpt4 but purely from the existing knowledge that it has got it has successfully managed to beat human beings in text annotation tasks do you know what is a text annotation task for example if you have to do NLP natural language processing or if you have to do something like text classification let's take this particular example you have a tweet and then you want to mark this tweet or flag this tweet whether this tweet supports Democratic party political affiliation or Republic political affiliation if you are if you are familiar with U.S political scene there are two major groups one is the Democratic party and then the second one is republic party and if you want to build a text classification system that classifies to eat whether a given tweet supports Democratic party or republic party then you need to have human annotators who will manually read these tweets and then Market oh this is democratic this is Republic demo Democratic Republic and then that data set is later on used to build the classifier that can help you classify the task this is how typically natural language processing works and human annotators play a very important role in fact in a lot of developing nations a lot of people actually do this data entry jobs The annotation job because all they do is they read something then Market whether it is human whether it is a political affiliation Republic political affiliation or Democratic political Affliction this is one of the examples of how annotation happens annotation happens at different levels but what we have seen in the latest paper is that chat GPT 4 has successfully outperformed like beaten human beings in terms of annotation task accuracy that is in zero short learning which means no additional data has been given to it and it has successfully managed to do it and that's what we are going to listen in this video where how this has happened what this has happened and what kind of implications that it might have the first thing for us to know is what is the task that is being discussed here this is a task of annotating political Twitter messages so what it says is charge apt4 outperforms experts and crowd workers in annotating political Twitter messages with zero short learning this paper was very recently published couple of days back so what is this paper doing it's assessing the accuracy reliability and bias of large language model for uh what is it doing it's doing the text analysis task of classifying the political affiliation of a Twitter poster based on the content of the Tweet so the llm is compared to manual annotation both by expert classifiers and also Cloud crowd workers generally considered to be golden standard crowd workers are golden standard because what would happen is a company that wants to build classification they will send the sticks to me they'll send this text to somebody else they'll send this text to probably 100 humans um signed up on Amazon Mechanical Turk or some other crowd platform and they will take all these things and they'll take the aggregate this is very similar like what they do with Ensemble models like when you have multiple machine learning models you ansible them together and then you take the majority of the answer because you trust them the same thing happens in The annotation world and what chat GPT or gpt4 has done here successfully is it managed to classify it much better than what human beings have been doing traditionally and if you see how it has been doing this chart is quite important here so you can see llm here the the blue color line that's where the chat GPT or GPT 4 power charge GB days and you can see the experts and you can see the combined mechanical deck like the human humans so technically whatever humans were doing previously chat GPT 4 has like done out perform completely it's not even you know closer line where you can say it's not statistically significant it has actually outperformed human being also whenever you are talking about political affiliation in fact like ticks classification one of the important things to keep in mind is the bias that people have usually some people have you know you look at a particular profile you think naturally that they are probably a democratic supporter you will get a profile you think they are a republic supported it's it's a it's an inherent bias that human beings usually have but if you look at the bias in and itself chat GPT also has a little bit of bias but it's it's it's probably the bias is lower than what the mechanical turkers had like if you see all coders are biased towards guessing Democrat over Republican the llms and experts are a similar level in bias but while mechanical took Amazon Mechanical classifiers have significantly stronger bias so even in terms of bias the chart GPT the charge GPT or AI bot here is almost close to what the experts are it is not like you know very heavily biased so this is a very interesting approach uh the very interesting um in fact advance in terms of how we are going to use charge GPT for example I'm kind of thinking about the economical implications I know a lot of people at least like I come from India a lot of people in India are employed at one task which is annotation task because Indian Resources are not very expensive like resources in the US so usually researchers from those countries they actually come to countries like India where they get cheap annotation resources so all you have to do is read something follow the instructions and then Mark it this is what people have been doing but now what charge GPT has done is it has managed to beat human beings that means I'm not sure how people would have people would be doing these kind of tasks going forward now how did they manage to do this with charge a bit like if you want to replicate this let's say you are a social scientist or you are a researcher and you want to do this for yourself how did you do this they first give this prompt to set a context and then give that particular role to charge GPT so what did they do so they have used API first of all they did not use the playground or the charge GPT interface that we typically use they used API and this is the instruction that has been given to charge GPD you will be given a set of Twitter posts from different U.S politicians sent during the two months preceding the 2020 U.S presidential election that is between September 3rd and November 3rd 2020. your task is to use your knowledge of U.S politics to make an U.S politics to make an educated guess on whether the poster is the Twitter or Twitter profile it's a Democrat or Republican respond either democrat or republican If the message does not have enough information for an educated guess just make your best guess this is the instruction that was given to charge GPT or gpt4 and this was used and the values are like they they ran the model five times at a lower temperature and higher temperature to capture the total variety of responses just to make sure you know it doesn't hallucinate a lot so typically you know that large language models are blamed for hallucination um it's it's quite a truth and also these days there is another thing that where they say that this is just a spitter like it's a beer spitter it just keeps on spitting some BS people make sense out of it and people use it you might ask why not use humans why don't you keep on using humans the problem is that humans while humans have been considered as the golden standard for these kind of tasks so there are several limitations one humans are inherently slow like we have to eat we have to sleep we have a lot of things to do in life we are costly which means missions like an APA token would definitely cost much cheaper than human being at least at this point and humans have this bias like I said you go to an average American citizen and ask them to do it they would have this bias but you cannot say modern will not have that bias models might also have that bias but probably lesser bias is what we have seen have limited attention span so you know that humans have this kind of disadvantages and that is where I think a system like charge GPD and gpt4 can completely topple what human beings can do and I see a future where people would probably use these kind of tools much better than human beings now while all of this is really good very impressive so far you have learned how chart GPT has beaten human beings quite an impressive thing I would like to bring to your attention one key element here I'm not sure how much of an issue that is but this is a very key thing whenever we discuss about charge GPT charged GPT is quite good at memorization whatever it has seen it has memorized that it has that knowledge inbuilt now what this analysis has done what this paper has done it has used Twitter messages from U.S politician during the 2020 election so it is completely possible that whatever chart GPT is being asked now is already part of charge gpd's training data I'm not sure what kind of annotation or what kind of instructions are gone before into that but the point here is that it is not completely new data that charge CPT has never seen before I would be quite impressed like more impressed if if this is let's say 2022 data and charge GPT has not seen this data and then we see similar result but you know there is a little bit of skepticism in me because this is a data that charge GPT has already seen it and that is while while I do not dispute the results of this paper I have that that a little bit of skepticism to say oh this data has been already shown to charge if it is somehow in some form so it's completely possible that it is really performing good better than human beings probably because it's already part of the training data I'm not sure but that is something that you need to keep in mind whenever you see these kind of papers that talks about oh charge repeat human means it has done great things then keep in mind that it's completely possible that there is some catch there but overall I think the the capability here is unquestionable you just set a context like we saw the instruction how to set a con context to charge APD like like this you will be given all these details and after you do that in a completely zero short learning center which means you don't give any additional examples chart GPT can really outperform human annotators in classifying political router messages I think human annotators are at the top of the job risk when chart GPT goes mainstream like a lot of people have when a lot of people start using it let me know in the comment section what do you think about chart GPT outperforming humans and what kind of political or economical implications it might have see in another video Happy prompting