SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
📰 ArXiv cs.AI
SciCoQA is a dataset for detecting discrepancies between scientific publications and their codebases
Action Steps
- Construct a dataset from GitHub issues and reproducibility papers
- Propose a synthetic data generation method for constructing paper-code discrepancies
- Analyze paper-code discrepancies in detail and propose discrepancy types and categories
Who Needs to Know This
Data scientists and machine learning researchers on a team can benefit from SciCoQA to ensure faithful implementations of their research, while software engineers can use it to improve the quality of their codebases
Key Insight
💡 Ensuring faithful implementations of scientific research is crucial for reproducibility and reliability
Share This
🚀 Introducing SciCoQA: a dataset for detecting discrepancies between scientific papers and codebases! 💻
Key Takeaways
SciCoQA is a dataset for detecting discrepancies between scientific publications and their codebases
Full Article
Title: SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
Abstract:
arXiv:2601.12910v2 Announce Type: replace-cross Abstract: We present SciCoQA, a dataset for detecting discrepancies between scientific publications and their codebases to ensure faithful implementations. We construct SciCoQA from GitHub issues and reproducibility papers, and to scale our dataset, we propose a synthetic data generation method for constructing paper-code discrepancies. We analyze the paper-code discrepancies in detail and propose discrepancy types and categories to better understa
Abstract:
arXiv:2601.12910v2 Announce Type: replace-cross Abstract: We present SciCoQA, a dataset for detecting discrepancies between scientific publications and their codebases to ensure faithful implementations. We construct SciCoQA from GitHub issues and reproducibility papers, and to scale our dataset, we propose a synthetic data generation method for constructing paper-code discrepancies. We analyze the paper-code discrepancies in detail and propose discrepancy types and categories to better understa
DeepCamp AI