When Terminal Agent Training Stalls

This paper presents a framework for automatically generating, validating, and evaluating terminal-based AI agent tasks at scale. Using thousands of tasks across multiple domains, it examines why agent training and evaluation can stagnate, highlighting the importance of distinguishing between agent limitations, task design issues, verifier errors, and infrastructure failures. The findings show that simply increasing task difficulty does not necessarily improve agent capabilities, while system-level factors can significantly affect benchmark results. The work emphasizes the need for reliable evaluation pipelines, robust verifiers, and transparent reporting practices to ensure that measured performance accurately reflects agent capabilities rather than artifacts of the training or execution environment.
https://d.dam.sap.com/x/im5nh5A?doi=SAP1361265