RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

📰 ArXiv cs.AI

Learn to evaluate code agents' repository context reasoning using RepoMirage, a two-stage evaluation suite

advanced Published 27 May 2026

Action Steps

Build a test suite using RepoMirage to evaluate code agents' repository context reasoning
Run perturbation experiments to analyze code agents' performance on end-to-end tasks
Configure the evaluation suite to test task-relevant information identification across multiple files
Test the code agents' ability to reason over relations among files
Apply the evaluation results to improve code agents' repository context reasoning capabilities

Who Needs to Know This

AI researchers and software engineers can benefit from this knowledge to improve code agents' performance on repository-level tasks

Key Insight

💡 RepoMirage is a two-stage evaluation suite that helps assess code agents' ability to identify task-relevant information and reason over relations among files

Key Takeaways

Learn to evaluate code agents' repository context reasoning using RepoMirage, a two-stage evaluation suite

Full Article

Title: RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

Abstract:
arXiv:2605.26177v1 Announce Type: cross Abstract: Code agents are currently having skillful performance on repository-level software engineering benchmarks, but it remains unclear whether success on end-to-end tasks such as issue resolution truly reflects repository context reasoning, the ability to identify the task-relevant information across multiple files and reason over the relations among them. To investigate this question, we introduce RepoMirage, a two-stage evaluation suite built on SWE

Read full paper → ← Back to Reads