GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

📰 ArXiv cs.AI

Learn how to identify execution bottlenecks in GUI vs. CLI agents and optimize computer-use agents' performance by understanding the differences in interaction modalities

advanced Published 24 Jun 2026

Action Steps

Configure a benchmarking environment to compare GUI and CLI agents
Run identical tasks on both GUI and CLI agents to identify execution bottlenecks
Analyze the results to determine the impact of interaction modality on task execution
Optimize agent performance by selecting the most suitable interaction modality for each task
Compare the performance of screen-only GUI agents and skill-mediated CLI agents

Who Needs to Know This

Software engineers, AI researchers, and DevOps teams can benefit from understanding the trade-offs between GUI and CLI agents to improve the efficiency of their computer-use agents

Key Insight

💡 Understanding the differences in execution bottlenecks between GUI and CLI agents can help optimize computer-use agents' performance

Key Takeaways

Learn how to identify execution bottlenecks in GUI vs. CLI agents and optimize computer-use agents' performance by understanding the differences in interaction modalities

Full Article

Title: GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

Abstract:
arXiv:2606.24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goa

Read full paper → ← Back to Reads