WideSWE: Can Coding Agents Manage Cross-Repository Change Coordination?

Meta description:
WideSWE tests coding agents on 120 cross-repository tasks from 103 software ecosystems, where best full success reaches 42.50% with Codex CLI and GPT-5.6-sol.

Tags: WideSWE, coding agents, cross-repository development, software benchmarks, AI coding evaluation

Featured image: abstract technology style, no real people, no company logos, no text

WideSWE introduces a benchmark that tests coding agents, meaning AI software-development systems that read repositories, write patches, and run tests, beyond isolated codebases and asks whether they can make changes that span multiple repositories. The setup matters because real engineering work often crosses package boundaries, service interfaces, and dependency graphs, so single-repo success can overstate practical capability. The benchmark contains 120 real-world tasks drawn from 103 software ecosystems, split evenly between 60 bug fixes and 60 features, and seven agent configurations score full task success between 10.83% and 42.50%.

From Single-Repo Tests to Ecosystem Reality

Coding-agent evaluation has shifted from resolving individual issues to carrying out longer development sequences, but the usual measuring stick remains a single codebase. That approach works for narrow repairs, yet many production changes require coordinated edits across repositories. A feature may need updates in an API service, a client library, and an integration test suite. A bug fix may require a shared dependency change and downstream callers to stay consistent. When tests only cover one repository, an agent can appear competent while missing the wider system contract.

Building WideSWE From 103 Software Ecosystems

The WideSWE authors mined and reviewed changes across 103 software ecosystems to construct tasks that preserve this cross-repository complexity. The resulting 120 tasks are balanced between 60 bug fixes and 60 features. Prompts are derived from related issues and pull requests, grounding the benchmark in real developer intent rather than synthetic prompts. Hidden tests, meaning evaluation tests not shown to the agent, are systematically reviewed and adapted so they can accept diverse correct implementations while still preserving required behavior and regression checks. This design gives evaluators a harder question: not whether an agent can patch a file, but whether it can complete a task that touches several repositories without breaking established behavior.

Latest Results: Agents Still Struggle Across Repositories

Across seven agent configurations, full task success ranges from 10.83% to 42.50%. The highest rate is achieved by the configuration pairing Codex CLI with GPT-5.6-sol. The spread shows that current coding agents can sometimes navigate cross-repository work, but reliable completion remains uncommon. Failure trajectories fall into three broad patterns. Some agents do not identify the necessary changes across repositories. Others recognize the required changes but leave the work unfinished. Still others modify the relevant repositories without fully satisfying the request, suggesting that awareness of scope and execution of scope are separate challenges.

A further comparison examines whether agents perform better when working on one repository at a time or when handling related repositories jointly. Under the same prompts, independent execution mainly helps recover omitted work. It is less effective at correcting changes that have already been attempted but remain unsuccessful. Joint execution can use information from related repositories to guide implementation and verification, pointing to a benchmark dimension that favors broader context rather than fragmented local fixes.

What WideSWE Suggests for the Next Evaluation Phase

The benchmark’s value is not simply a new score for existing agents. It reframes the evaluation question around software ecosystems, where correctness depends on coordination across multiple repositories. The current numbers place WideSWE in a diagnostic stage: agents can solve some cross-repository tasks, but the gap between 10.83% and 42.50% full success shows that completion is still uneven. The code is available at the referenced https URL, giving other teams a basis for reproducing the task set and comparing new agent designs. What to watch next is how future systems handle shared interfaces, dependency updates, and cross-repository verification without breaking existing behavior.