Few Shot Code Generation to Autonomous Software Engineering Agents // John Yang

//Abstract Software Engineering can serve as a diverse, rich testbed for evaluating the next generation of language models and driving their development. This talk introduces a line of three works that has established the potential of this research direction and guided industry advancements towards autonomous software engineers. First, SWE-bench is a benchmark that evaluates an Al system's capability to resolve real world GitHub issues, featuring 2294 task instances collected from 12 distinct Python repositories. Second, SWE-agent is an autonomous system that uses a language model to interact with a computer to solve software engineering tasks, setting a state-of-the-art 12.5% resolved rate on the SWE-bench test set. Lastly, SWE-bench Multimodal, a new dataset of 617 task instances from 17 JavaScript repositories, shows how many existing coding agents are overfit to Python, raising the property of "generalizability as an overlooked but desirable trait of AI systems. //Bio John Yang is a 1st year PhD student at Stanford University advised by Professor Diyi Yang. Previously, he was a Master's student at Princeton University, where he was advised by Karthik Narasimhan. John's research directions include Language Agents, Language Model Evaluation, and Software Engineering. A Prosus | MLOps Community Production