By changing just one line of code, Google's AI agent optimised Harness, resulting in an eightfold increase in task success rate.只改一处代码,谷歌 AI 智能体优化 Harness 后任务成功率提升 8 倍
According to IT Home on 20 November, Google researchers published a new research paper titled "Harnesses That Improve Themselves" on 17 November (local time)…
IT之家 11 月 20 日消息,谷歌研究人员当地时间 11 月 17 日最新发表的研究论文《Harnesses That Improve Themselves》(《能够自我改进的 Harness》)…

Key Points要点速览
According to IT Home on 20 November, the latest research paper published by Google researchers on 17 November (local time), "Harnesses That Improve Themselves", shows that by allowing AI agents to gradually modify their own operating frameworks and validate each modification, the completion rate of software engineering tasks can be significantly improved. In a SWE-bench Lite test, the success rate of agent tasks using a modular optimisation approach increased from 5.0% to 40.0%, eight times the original rate, with an average of approximately 46,700 tokens consumed per task. The research focuses on the Harness of AI agents. It is not the underlying AI model itself, but rather an operating framework built around the model, responsible for managing context and memory, controlling the execution of multi-turn tasks, calling external tools, and verifying the execution results before the task is completed. The researchers pointed out that different models may exhibit significantly different weaknesses in the way they use tools and in the process of executing tasks. Therefore, a Harness optimised for one model may experience issues such as task stagnation, excessive token consumption, or reduced execution performance when migrated to other models. To address this issue, the researchers proposed a self-improving Harness architecture. Unlike the traditional approach of constraining agents by modifying prompts, this solution allows the optimisation model to directly modify Harness's code and decide whether to retain the modifications based on testing. The experiment used an initial programming agent, Harness, consisting of only 142 lines of code, which was divided into four editable modules responsible for context and state management, agent control, tool call interfaces, and validation and fault recovery, respectively. During each round of optimisation, the system only allows modifications to one of the modules. The optimisation model first identifies problems based on the agent's execution records and proposes code modification solutions. It then checks the effects of the modifications using a validation set, while being constrained by the token usage limit. Only modifications that pass validation are retained, while solutions that do not meet the requirements are discarded. In the core test, the researchers used Gemini 3.8 Flash as the base model for executing tasks and evaluated the performance of different solutions on 40 SWE-bench Lite software engineering tasks that were not involved in the optimisation process. The results showed that the initial Harness had a task success rate of 5. 0%. After adopting the modular modification and verification mechanism, the success rate increased to 40.0%, with an average of approximately 46,700 tokens consumed per task. For comparison, if the optimised model is allowed to rewrite Harness in its entirety without restriction, the task success rate can reach 42.5%, which is slightly higher than the modular approach, but each task consumes an average of approximately 451,600 tokens, nearly 10 times that of the modular approach. IT Home noted that another experiment constrained the agent by modifying only the prompt rule file, resulting in a drop in the task success rate from the initial 5.0% to 2.5%. The study also examined the performance of the optimised Harness when migrated to other models. Without performing an optimisation search again, after migrating the modular approach to Gemini 4 Argon, the task success rate reached 37.5%, while the complete rewrite approach achieved 30.0%. The researchers believe that Harness's self-improvement should not rely solely on prompts, but should instead implement important constraints in the actual code. For example, by using automated tests to check the results of task execution, identify repeated failures, and prevent unvalidated tasks from ending, agents can complete complex tasks more reliably. The paper also points out that allowing agents to modify their own running code also introduces new risks, including the continuous expansion of prompt content, over-optimisation for benchmark tests, and tampering with evaluation procedures. Therefore, code modifications need to be constrained by verification mechanisms and permission boundaries. It should be noted that the experiments in this study were mainly conducted on specific models and SWE-bench Lite software engineering tasks, and the core test set contained only 40 tasks that were not optimised. This is insufficient to demonstrate that the method can achieve the same results across all models and agent application scenarios.
IT之家 11 月 20 日消息,谷歌研究人员当地时间 11 月 17 日最新发表的研究论文《Harnesses That Improve Themselves》(《能够自我改进的 Harness》)显示,通过让 AI 智能体逐步修改自身运行框架,并对每次修改进行验证,可以显著提高软件工程任务的完成率。 在一项 SWE-bench Lite 测试中,采用模块化优化方案的智能体任务成功率从 5.0% 提升至 40.0%,达到原来的 8 倍,平均每项任务消耗约 4.67 万 Token。 其研究重点是 AI 智能体的 Harness。它并非底层 AI 模型本身,而是围绕模型构建的运行框架,负责管理上下文和记忆、控制多轮任务执行、调用外部工具,以及在任务结束前验证执行结果。 研究人员指出,不同模型在工具使用方式和任务执行过程中表现出的弱点可能存在明显差异。因此,为某个模型优化的 Harness,迁移到其他模型后可能出现任务停滞、Token 消耗过高或执行效果下降等问题。 为解决这一问题,研究人员提出了一套能够自我改进的 Harness 架构。与传统上通过修改提示词来约束智能体不同,该方案允许优化模型直接修改 Harness 的代码,并通过测试决定是否保留修改。 实验采用一套仅有 142 行代码的初始编程智能体 Harness,将其拆分为四个可编辑模块,分别负责上下文与状态管理、智能体控制、工具调用接口,以及验证与故障恢复。 每轮优化过程中,系统只允许修改其中一个模块。优化模型首先根据智能体执行记录识别问题并提出代码修改方案,随后通过验证集检查修改效果,同时受到 Token 使用上限约束。只有通过验证的修改才会被保留,未达到要求的方案则会被舍弃。 在核心测试中,研究人员使用 Gemini 3.8 Flash 作为执行任务的基础模型,并在 40 道未参与优化的 SWE-bench Lite 软件工程任务上评估不同方案的表现。 结果显示,初始 Harness 的任务成功率为 5.0%,采用模块化修改和验证机制后,成功率提高至 40.0%,平均每项任务消耗约 4.67 万 Token。 作为对照,如果允许优化模型不受限制地整体重写 Harness,任务成功率可以达到 42.5%,略高于模块化方案,但平均每项任务需要消耗约 45.16 万 Token,是模块化方案的近 10 倍。 IT之家注意到,另一项实验则采用仅修改提示词规则文件的方式约束智能体,结果任务成功率反而从初始的 5.0% 降至 2.5%。 研究还考察了优化后的 Harness 迁移到其他模型时的表现。在不重新进行优化搜索的情况下,模块化方案迁移至 Gemini 4 Argon 后,任务成功率达到 37.5%,而整体重写方案为 30.0%。 研究人员认为,Harness 的自我改进不应仅依赖提示词,而应将重要约束落实到实际代码中。例如,通过自动测试检查任务执行结果、识别重复失败和阻止未经验证的任务结束,可以让智能体更可靠地完成复杂任务。 论文同时指出,允许智能体修改自身运行代码也会带来新的风险,包括提示词内容不断膨胀、针对基准测试进行过度优化,以及篡改评估程序等。因此,代码修改需要受到验证机制和权限边界约束。 需要注意的是,这项研究的实验主要围绕特定模型和 SWE-bench Lite 软件工程任务展开,核心测试集仅包含 40 道未参与优化的任务,尚不足以证明该方法在所有模型和智能体应用场景中都能取得相同效果。
想马上用起来?去「AI工具」栏挑一个直接下载,或在「AI教程」里跟着图文步骤做一遍。
去 AI工具 → 看 AI教程 →阅读与点赞数据保存在你的浏览器本地,欢迎留下你的想法。
今日正能量学一点,用一点;今天种下的种子,会长成明天的能力。去免费下载专区 →广告
评论 文明发言,让讨论更有价值