Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators

arXiv:2503.06778v3 Announce Type: replace-cross Abstract: Event annotation is important for identifying market changes, monitoring breaking news, and understanding sociological trends. Although expert annotators set the gold standards, human coding is expensive and inefficient. Unlike information extraction experiments that focus on single contexts, we evaluate a holistic workflow that removes irrelevant documents, merges documents about the same event, and annotates the events. Although LLM-bas