Probe-and-refine tuning lifts coding agent resolve rate to 33%
A new procedure called probe-and-refine tuning iteratively improves `AGENTS.md` guidance files using synthetic bug-fix probes, lifting a coding agent's SWE-bench Verified resolve rate from 25.5% (unguided) to 33.0%.
Score breakdown
The paper resolves a contested debate by showing that guidance production method — not guidance presence alone — determines whether `AGENTS.md` files help or hurt coding agents, and provides a concrete tuning procedure that raises SWE-bench Verified resolve rate by 7.5 percentage points over an unguided baseline.
- 01Probe-and-refine tuning uses synthetic bug-fix probes and single-shot LLM calls to iteratively improve repository guidance files, with no agent loop or tool use during tuning.
- 02On SWE-bench Verified with Qwen3.5-35B-A3B at 200 steps (4 trials), probe-and-refine achieves a 33.0% mean resolve rate.
- 03The static knowledge base baseline scores 28.3% and the unguided baseline scores 25.5%; both contrasts are significant at p < 0.001.
Asa Shepard and Jeannie Albrecht address a contested question in agentic coding research: do `AGENTS.md` guidance files actually help LLM-based coding agents? Prior studies disagree on whether LLM-generated guidance improves or harms performance. The paper argues that the method of producing guidance is the decisive variable, and introduces probe-and-refine tuning — a procedure that runs synthetic bug-fix probes against a repository, then uses single-shot LLM calls to iteratively diagnose and patch the guidance file, without any agent loop or tool use during the tuning phase itself.
This indicates that better guidance helps agents navigate to the correct file rather than improving the quality of edits once there.
On SWE-bench Verified, probe-and-refine tuning with Qwen3.5-35B-A3B at 200 steps achieves a 33.0% mean resolve rate across four independent trials, compared to 28.3% for the static knowledge base used to initialize it and 25.5% for an unguided baseline (p < 0.001 for both contrasts). The improvement is attributed to coverage rather than precision: refined guidance causes agents to produce evaluable patches for 14.5 percentage points more instances, while per-patch precision remains statistically constant at approximately 59% (p = 0.119). This indicates that better guidance helps agents navigate to the correct file rather than improving the quality of edits once there.
Two additional experiments illuminate the method's boundaries. A step-budget experiment shows that guidance is what enables an agent to use a larger step budget productively — without it, extra steps do not translate into better outcomes. A cross-model experiment using NVIDIA-Nemotron-3-Nano-30B-A3B finds that the tuning loop degrades when the model cannot generate sufficiently diagnostic output, though per-patch precision remains constant even in that degraded setting.
Key facts
- 01Probe-and-refine tuning uses synthetic bug-fix probes and single-shot LLM calls to iteratively improve repository guidance files, with no agent loop or tool use during tuning.
- 02On SWE-bench Verified with Qwen3.5-35B-A3B at 200 steps (4 trials), probe-and-refine achieves a 33.0% mean resolve rate.
- 03The static knowledge base baseline scores 28.3% and the unguided baseline scores 25.5%; both contrasts are significant at p < 0.001.
- 04The gain is driven by coverage: refined guidance produces evaluable patches for 14.5 percentage points more instances.
- 05Per-patch precision remains statistically constant at ~59% (p = 0.119), meaning guidance helps agents reach the right file, not improve edit quality.
- 06A step-budget experiment shows guidance is what allows agents to use a larger step budget productively.
- 07A cross-model test with NVIDIA-Nemotron-3-Nano-30B-A3B finds the tuning loop degrades when the model cannot generate sufficiently diagnostic output.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 19, 2026 · 10:25 UTC. How this works →