UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
📰 ArXiv cs.AI
UI-Voyager is a self-evolving GUI agent that learns from failed experiences using Rejection Fine-Tuning and Evolutionary Exploration
Action Steps
- Employ Rejection Fine-Tuning (RFT) to learn from failed trajectories
- Use Evolutionary Exploration to adapt to changing GUI environments
- Evaluate the performance of the GUI agent using sparse rewards
- Refine the agent's policy using the learned experiences
Who Needs to Know This
AI engineers and researchers working on multimodal large language models and GUI agents can benefit from this approach to improve the efficiency of learning from failed trajectories and sparse rewards
Key Insight
💡 Learning from failed experiences can improve the efficiency of GUI agents in long-horizon tasks
Share This
🤖 UI-Voyager: a self-evolving GUI agent that learns from failed experiences! 📈
Key Takeaways
UI-Voyager is a self-evolving GUI agent that learns from failed experiences using Rejection Fine-Tuning and Evolutionary Exploration
Full Article
Title: UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
Abstract:
arXiv:2603.24533v1 Announce Type: cross Abstract: Autonomous mobile GUI agents have attracted increasing attention along with the advancement of Multimodal Large Language Models (MLLMs). However, existing methods still suffer from inefficient learning from failed trajectories and ambiguous credit assignment under sparse rewards for long-horizon GUI tasks. To that end, we propose UI-Voyager, a novel two-stage self-evolving mobile GUI agent. In the first stage, we employ Rejection Fine-Tuning (RFT
Abstract:
arXiv:2603.24533v1 Announce Type: cross Abstract: Autonomous mobile GUI agents have attracted increasing attention along with the advancement of Multimodal Large Language Models (MLLMs). However, existing methods still suffer from inefficient learning from failed trajectories and ambiguous credit assignment under sparse rewards for long-horizon GUI tasks. To that end, we propose UI-Voyager, a novel two-stage self-evolving mobile GUI agent. In the first stage, we employ Rejection Fine-Tuning (RFT
DeepCamp AI