Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs
📰 ArXiv cs.AI
Learn how to evaluate LLM variability in evidence screening for software engineering systematic literature reviews and improve study screening accuracy
Action Steps
- Apply LLMs to evidence screening in software engineering SLRs to assess their performance
- Compare the results of different LLMs to identify variability and potential biases
- Evaluate the trade-offs between accuracy, cost, and risk in study screening using LLMs
- Configure LLMs to optimize their performance in evidence screening tasks
- Test the robustness of LLMs in handling inconsistent or incomplete data in study screening
Who Needs to Know This
Researchers and software engineers on a team conducting systematic literature reviews can benefit from understanding LLM variability to improve the accuracy and efficiency of their study screening process
Key Insight
💡 LLM variability can significantly impact the accuracy and reliability of evidence screening in software engineering systematic literature reviews
Share This
🤖 LLMs can improve study screening in software engineering SLRs, but variability between models can impact accuracy. Learn how to evaluate and optimize LLM performance 📊
Key Takeaways
Learn how to evaluate LLM variability in evidence screening for software engineering systematic literature reviews and improve study screening accuracy
Full Article
Title: Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs
Abstract:
arXiv:2604.27006v1 Announce Type: cross Abstract: Context: Study screening in systematic literature reviews is costly, inconsistency-prone, and risk-asymmetric, since false negatives can compromise validity. Despite rapid uptake of Large Language Models (LLMs), there is limited evidence on how such models behave during the study screening phase, particularly regarding the choice of specific LLMs and their comparison with classical models. Objective: To assess LLM performance and variability in s
Abstract:
arXiv:2604.27006v1 Announce Type: cross Abstract: Context: Study screening in systematic literature reviews is costly, inconsistency-prone, and risk-asymmetric, since false negatives can compromise validity. Despite rapid uptake of Large Language Models (LLMs), there is limited evidence on how such models behave during the study screening phase, particularly regarding the choice of specific LLMs and their comparison with classical models. Objective: To assess LLM performance and variability in s
DeepCamp AI