Every processed story in chronological order, with the newest coverage first. Filter by tag, source, or score to drill in.
ReproRepo replaces the manual curation bottleneck of prior reproducibility benchmarks with a scalable, naturally occurring signal from GitHub issues, enabling ongoing large-scale evaluation of LLM agents on real-world ML paper auditing.
Because any single harness component can move benchmark scores by margins comparable to those between adjacent model generations, end-to-end scores can misattribute performance gains and mislead practitioners trying to improve agentic systems.
The paper identifies task decomposition — not retrieval — as the binding constraint in multi-skill agent planning, and SAD's single-iteration fix raises decomposition accuracy by over 32 percentage points, directly improving how reliably agents can assemble executable plans from large real-world skill libraries.
The finding that 80.2% of agent-authored test patches lack meaningful assertions means that quality gates relying on test-file presence give a false signal of verification coverage in AI-generated code.
The paper establishes that PLT performance saturates at exactly two loops and provides a gain–cost diagnostic framework explaining why, giving practitioners a principled basis for loop-count selection rather than relying on monotonic scaling assumptions.
The finding that non-software occupations achieve success rates within 7 percentage points of software engineering on Claude Code's strictest metric suggests the tool's effectiveness is not limited to developers.
The framework demonstrates that automated prompt optimization alone — without any fine-tuning — can turn a completely failing LLM agent (0% on PutNext) into one that succeeds nearly three-quarters of the time, showing prompt engineering can be systematically automated rather than done by hand.
The paper demonstrates that source attribution is an independent axis of factuality verification — meaning standard source-blind metrics can pass answers that contain incorrect attributions, a gap ProvenanceGuard is designed to close in MCP-based agents.
The benchmark exposes concrete, measurable gaps in LLM agents' ability to infer hidden world models through interaction, providing a rigorous testbed with classical algorithm baselines that quantifies how far current agents fall short of robust interactive discovery.
The study establishes that explicit delegation contracts improve the reviewability of AI coding agent work — not its correctness — reframing the contract as a mechanism for human oversight rather than a driver of agent task performance.