FVSpec: Real-World Property-Based Tests as Lean Challenges
📰 ArXiv cs.AI
Learn how to apply property-based tests to real-world software verification tasks using FVSpec, a benchmark for evaluating AI models and agents
Action Steps
- Scrape property-based tests from real-world Python repositories using tools like GitHub API
- Translate property-based tests into Lean 4 specifications with sorry placeholders using automated translation tools
- Evaluate AI models and agents on the translated Lean specifications using metrics like quality and accuracy
- Compare the performance of different AI models and agents on the FVSpec benchmark
- Apply the insights gained from the benchmark to improve the development of more robust and reliable software systems
Who Needs to Know This
This benchmark is useful for AI researchers, software engineers, and verification experts who want to evaluate and improve the performance of AI models and agents on real-world formal software verification tasks. It can help teams develop more robust and reliable software systems.
Key Insight
💡 Property-based tests can be effectively translated into Lean specifications to evaluate AI models and agents on real-world software verification tasks
Share This
🚀 Introducing FVSpec: a benchmark for evaluating AI models and agents on real-world formal software verification tasks 🚀
Key Takeaways
Learn how to apply property-based tests to real-world software verification tasks using FVSpec, a benchmark for evaluating AI models and agents
Full Article
Title: FVSpec: Real-World Property-Based Tests as Lean Challenges
Abstract:
arXiv:2606.01008v1 Announce Type: cross Abstract: We present a benchmark for evaluating AI models and agents on real-world formal software verification tasks. We first scrape 11,039 property-based tests (PBTs) from real-world Python repositories, then automatically translate 2,772 of them (25%) into 9,415 Lean 4 specifications with sorry placeholders (about 3 formalizations/PBT; we retain multiple attempts when none dominates on quality metrics). Translating PBTs into Lean specifications is chal
Abstract:
arXiv:2606.01008v1 Announce Type: cross Abstract: We present a benchmark for evaluating AI models and agents on real-world formal software verification tasks. We first scrape 11,039 property-based tests (PBTs) from real-world Python repositories, then automatically translate 2,772 of them (25%) into 9,415 Lean 4 specifications with sorry placeholders (about 3 formalizations/PBT; we retain multiple attempts when none dominates on quality metrics). Translating PBTs into Lean specifications is chal
DeepCamp AI