docs: evaluate the actual Darwinian evolver + recast README around the promise - #135
Merged
Conversation
…est-of-N The imbue-ai/darwinian_evolver, wired to our gate (oracle) and proposer (mutator) and run at equal budget against best-of-N on the 23 bugs the single-proposer loop fails: 6/23 vs 8/23 robust recovery, zero recoveries best-of-N missed, and its learning-log/verification levers inert (controlled == best-shot). Confirms the earlier proxy result on the real tool. The value is the signal plus the gate, not the search; a search engine lifts neither the verification nor the supply constraint on autonomous self-improvement.
The opening led with the gate and a self-questioning hedge; recast it as the value — automated evolution of the skills, prompts, and tool code an agent runs on, with the gate as the proof that each change is real. Fold the no-GPU/cost note into the lead and promote the framework-compatibility line. Update the Findings section to cite the actual imbue darwinian_evolver evaluation (6/23 vs 8/23, strictly dominated) in place of the earlier proxy, and link the report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
reports/darwinian_evolver_evaluation.md(with companion result datareports/darwinian_evolver_evaluation.json): a pre-registered, equal-budget evaluation of the actual imbuedarwinian_evolveragainst best-of-N resampling, on the 23 bugs the single-proposer loop fails.Finding
Best-of-N recovered 8/23; the evolver 6/23. The evolver recovered nothing best-of-N missed (evolver-only = 0), and its distinguishing machinery — learning log, post-mutation verification, weighted parent selection — was inert: the controlled and best-shot arms are identical. Budgets actually favored the evolver (~22-25% more proposer calls, because best-of-N stops on first deploy) and it still lost, so the comparison is conservative. This confirms the earlier proxy result on the real tool.
The value is the signal plus the gate, not the search. A search engine lifts neither the verification constraint (the gate and the upstream-fix test are the fitness function — a self-directed loop would have to author its own oracle) nor the supply constraint (the bug feed is effectively empty across the harvested corpus) that bound autonomous self-improvement.
Hygiene
The AGPL tool is imported only in an isolated, unshipped evaluation harness — never into the framework's MIT modules, and added to the environment only at run time. Nothing AGPL is committed: the report cites the tool by URL and ships only prose plus result data.