SWE-Future uses repo forecasts to synthesize future-oriented coding benchmarks

Qiao Zhao, JianYing Qu, and Jun Zhang introduce SWE-Future, a forecast-conditioned data synthesis method designed to address a core vulnerability in coding-agent benchmarks: the tendency to replay public GitHub issues and pull requests, which exposes benchmarks to overlap with model pretraining, fine-tuning, synthetic-data generation, or benchmark-driven model selection. While fully synthetic tasks avoid this replay problem, they risk drifting away from real repository needs. SWE-Future threads this needle by forecasting future repository work — covering feature implementation/enhancement, bugfix, and refactor task families — using only evidence available before a fixed snapshot time T₀.

The forecasting step is validated retrospectively: forecasts are fixed at T₀, and later pull requests are used solely to measure whether the predicted task families match actual future repository work.

The forecasting step is validated retrospectively: forecasts are fixed at T₀, and later pull requests are used solely to measure whether the predicted task families match actual future repository work. Across an 80-repository study, the forecaster achieves 58.1% future-work relevance under the main semantic matching metric. The validated forecast families then serve as conditioning signals to synthesize a 200-task coding-agent dataset spanning 61 repositories, drawn from a task-generation snapshot rather than the later pull requests used for validation. The paper argues that repository-evolution forecasting can guide realistic, future-oriented coding-task synthesis while reducing direct dependence on historical pull-request replay.

Key facts

1. 01SWE-Future is a forecast-conditioned data synthesis method for future-oriented coding-agent benchmarks.
2. 02The method forecasts future feature implementation/enhancement, bugfix, and refactor task families using only pre-T₀ repository evidence.
3. 03Forecasting was validated retrospectively across an 80-repository study.
4. 04The forecaster achieved 58.1% future-work relevance under the main semantic matching metric.
5. 05Validated forecast families were used as conditioning signals to synthesize a 200-task coding-agent dataset.
6. 06The dataset spans 61 repositories and avoids replaying the later pull requests used for validation.
7. 07The approach reduces direct dependence on historical pull-request replay, mitigating data-contamination risks.

Topics