Instruction files for AI coding agents don't reliably improve PR merge rates
A study of 15,549 agentic PRs across 148 projects finds that adding instruction files for AI agents like GitHub Copilot produces mixed results, with roughly equal shares of projects seeing merge rates rise or fall by at least 20%.
Score breakdown
The study shows that simply adding instruction files for AI coding agents does not guarantee better pull request outcomes, and that file length and structure appear to be differentiating factors — motivating a new research direction around treating instruction file authorship as a disciplined engineering practice.
- 01Study analyzed 15,549 agentic PRs from 148 projects in the AIDev dataset
- 02Instruction files guide AI agents on project navigation, testing, and best practices
- 0327.7% of projects increased merge rate by at least 20% after adding instruction files
Ali Arabat and Mohammed Sayagh investigated whether instruction files — used to guide AI coding agents like GitHub Copilot on how to navigate a project, locate components, run tests, and follow best practices — actually lead to better agentic pull requests. Using the AIDev dataset, they analyzed 15,549 agentic PRs from 148 projects, comparing each project's performance before and after instruction files were created across three dimensions: merge rate (as a proxy for PR success), code churn and number of modified files (as proxies for task complexity), and time to merge and number of comments (as proxies for merge effort).
The results show that instruction files do not reliably improve agent performance.
The results show that instruction files do not reliably improve agent performance. While 27.7% of projects increased their merge rate by at least 20% after introducing instruction files, 26.35% experienced a decrease of similar magnitude. The same inconsistency appeared across the complexity and effort dimensions. An initial exploratory analysis, however, found a distinguishing characteristic among projects that did improve: their instruction files were substantially longer and more structured, organized into a higher number of sections and sub-sections.
The authors frame these findings as motivation for a new research direction they call "Instructions-as-Code" — treating the creation and maintenance of agent instruction files as a disciplined software engineering activity, rather than an informal or ad hoc practice. The study draws on the AIDev dataset and focuses specifically on the before-and-after impact of instruction file creation within individual projects.
Key facts
- 01Study analyzed 15,549 agentic PRs from 148 projects in the AIDev dataset
- 02Instruction files guide AI agents on project navigation, testing, and best practices
- 0327.7% of projects increased merge rate by at least 20% after adding instruction files
- 0426.35% of projects decreased their merge rate after adding instruction files
- 05Three measurement dimensions: merge rate, code churn/modified files, and time to merge/number of comments
- 06Projects with improved merge rates had substantially longer, more structured instruction files with more sections and sub-sections
- 07Authors propose treating instruction file development as a formal software engineering activity called 'Instructions-as-Code'
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 12, 2026 · 10:05 UTC. How this works →