Grounding Computer Use Agents on Human Demonstrations
📰 ArXiv cs.AI
Learn how to ground computer use agents on human demonstrations for reliable natural language instruction following
Action Steps
- Collect human demonstration data for desktop environments using tools like screen recording software
- Build a large-scale dataset like GroundCUA to cover various applications and categories
- Use the dataset to train and fine-tune machine learning models for grounding computer use agents
- Evaluate the performance of the agents using metrics like accuracy and precision
- Apply the grounded agents to real-world scenarios to test their reliability and effectiveness
Who Needs to Know This
AI researchers and engineers working on human-computer interaction and natural language processing can benefit from this research to improve the accuracy of their agents
Key Insight
💡 Grounding computer use agents on human demonstrations is crucial for accurate natural language instruction following
Share This
🤖 Ground computer use agents on human demos to improve NL instruction following! 📊
Key Takeaways
Learn how to ground computer use agents on human demonstrations for reliable natural language instruction following
Full Article
Title: Grounding Computer Use Agents on Human Demonstrations
Abstract:
arXiv:2511.07332v2 Announce Type: replace-cross Abstract: Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GroundCUA, a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and
Abstract:
arXiv:2511.07332v2 Announce Type: replace-cross Abstract: Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GroundCUA, a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and
DeepCamp AI