LLMs Do Not Grade Essays Like Humans

1. 1Evaluate the performance of LLMs in automated essay scoring using datasets with human-graded essays
2. 2Compare the scores generated by LLMs with human grades to assess agreement
3. 3Analyze the grading behavior of different LLM models, such as GPT and Llama, in an out-of-the-box setting
4. 4Investigate the reasons for the weak agreement between LLM and human scores, such as differences in grading criteria or bias in the models

AI engineers and educators can benefit from understanding the limitations of LLMs in automated essay scoring, as it can inform the development of more accurate grading tools

💡 LLMs have limited ability to mimic human grading behavior, highlighting the need for further research and development in automated essay scoring

📝 LLMs struggle to grade essays like humans, with weak agreement between machine and human scores