TejasviTomar
October 2026 · Preprint

Do Self-Improving Coding Agents Beat Textbook Baselines? Evidence from a Darwin Gödel Machine Port and Two Published Systems

Tejasvi Singh Tomar

In plain words

The Darwin Gödel Machine evolves a coding agent that rewrites its instructions and tools. The machine's creators reported that this raised the agent's success rate on real GitHub issues from 20% to 50%. This paper reran the machine's best agent and its starting version on other GitHub issues, using a newer language model. The best agent did no better than the starting version. The paper's main experiments evolved agents for hard programming-contest problems. With one particular language model, a standard change (letting the model write longer replies) worked nearly as well as the best of these agents.

Abstract

Self-improving agent harnesses such as the Darwin Gödel Machine (DGM) let a coding agent edit its own scaffolding, and they keep the edits that score better and report large gains on coding benchmarks. In three pre-registered rounds, we compare what such loops find with standard engineering changes ("textbook moves"), one-shot redesigns by a language model, and placebo loops that make no edits; we also re-run released agents of the DGM and of ADAS with newer models. On hard LiveCodeBench problems with Claude Haiku 5.5, the starting agent often stops on an empty turn because its output-token limit is too small for the model's reasoning. A single textbook move, which raises this limit to 16,000 tokens and makes three other changes to the model loop, recovers about 90% of the gain of the best loop agent and costs less than half as much per task. Four of five one-shot redesigns by Claude Opus 5.5 also raise the limit, to 8,192 tokens, without evaluation feedback. The best loop agent still leads the textbook moves by 1–5 percentage points in pass rate. On held-out SWE-bench Verified tasks, run in the DGM's harness, the DGM's released best agent does not outperform its initial agent; three textbook moves on the initial agent outperform the best agent at a quarter of its cost, whether or not the patches of runs that ended early, most of them the best agent's, are kept. Claims of self-improvement should be measured against textbook moves, one-shot redesigns, placebo loops, held-out tasks and newer models.

Two charts side by side. Left, titled (a) reported progress: best selection score in percent against generation, from 0 to 40. Three blue step lines for DGM loops end at 74, 76 and 88%. Three orange step lines for placebo loops end at 68 to 70%, close to a dashed curve for the expected maximum of placebo scores, which rises to about 69%. A grey band from about 60 to 65% marks the seed's mean plus or minus one standard deviation. Right, titled (b) fresh problems vs cost: fresh pass rate in percent against cost per thousand tasks in US dollars. A blue curve for the DGM seed, best of k, rises from 65.6% at $3.9 to 73.9% at $13.6, and an orange curve for the engineered seed, best of k, rises from 86% at about $9 to 89% at about $23. Green diamonds for the evolved agents (loop bests) lie at about 75 and 77% for loops 1 and 2, and between 87 and 89% for loop 3, the no-hint loops and the engineered-seed loops.
Figure 1: First-round loops. (a) Best selection score by generation. Placebo loops (children are exact copies) rise to the expected maximum of their own noisy scores (dashed); DGM loops rise above it. (b) Pass rate on F₁ against cost per task. Lines: the unchanged seeds run k times, submitting the first solution that passes the examples in the problem statement. Diamonds: best agents of the loops.