Constructing Evaluation Datasets for Procedural Reasoning: Balancing Naturalness, Grounding, and Multi-Hop Coverage
📰 ArXiv cs.AI
Learn to construct evaluation datasets for procedural reasoning in AI-supported learning systems, balancing naturalness, grounding, and multi-hop coverage
Action Steps
- Construct a TMK model to represent the instructional knowledge
- Generate questions using strict TMK-based generation strategies
- Compare the quality of datasets generated using different strategies, including transcript-first generation with post-hoc TMK
- Evaluate the naturalness, grounding, and multi-hop coverage of the generated datasets
- Refine the dataset construction process based on the evaluation results
Who Needs to Know This
AI researchers and developers building evaluation datasets for procedural reasoning tasks can benefit from this knowledge to improve the quality of their datasets and ultimately the performance of their AI systems
Key Insight
💡 Balancing naturalness, grounding, and multi-hop coverage is crucial for constructing effective evaluation datasets for procedural reasoning
Share This
🤖 Construct high-quality evaluation datasets for procedural reasoning in AI-supported learning systems 📚
Key Takeaways
Learn to construct evaluation datasets for procedural reasoning in AI-supported learning systems, balancing naturalness, grounding, and multi-hop coverage
Full Article
Title: Constructing Evaluation Datasets for Procedural Reasoning: Balancing Naturalness, Grounding, and Multi-Hop Coverage
Abstract:
arXiv:2606.12767v1 Announce Type: new Abstract: Evaluating procedural reasoning in AI-supported learning systems requires question-answer datasets that are both learner-like and grounded in the instructional knowledge the system is expected to use. We study how TMK-based question generation strategies affect dataset quality for procedural and multi-hop reasoning. We compare three strategies: strict generation from Task-Method-Knowledge (TMK) models, transcript-first generation with post-hoc TMK
Abstract:
arXiv:2606.12767v1 Announce Type: new Abstract: Evaluating procedural reasoning in AI-supported learning systems requires question-answer datasets that are both learner-like and grounded in the instructional knowledge the system is expected to use. We study how TMK-based question generation strategies affect dataset quality for procedural and multi-hop reasoning. We compare three strategies: strict generation from Task-Method-Knowledge (TMK) models, transcript-first generation with post-hoc TMK
DeepCamp AI