Add a native cross-phase evolution orchestrator - #145
Merged
Conversation
Sequence the per-subsystem evolvers (skills -> tools -> prompts -> code) from one YAML run-spec instead of invoking each by hand. `python -m evolution.orchestrator` runs each phase as an isolated subprocess, captures its existing gate_decision.json verdict at a deterministic --output-dir, and writes a JSONL run history + summary. Propose-only by construction: it never deploys, and strips every per-phase create_pr unless --allow-pr is passed (a surviving PR phase must carry its spend ceiling). Verdict status is grounded in the gate file, not the evolvers' inconsistent exit codes -- a clean run whose gate rejected the candidate is a pass, not a failure. Fault isolation is the subprocess boundary; continue-on-error by default, --stop-on-error to halt, --resume to skip phases already in the ledger. Also adds the missing --output-dir CLI option to the skills and tools evolvers (their evolve() already accepted it) so the orchestrator captures every phase's run dir uniformly. Resolves our triage item for the upstream cross-phase orchestrator (NousResearch#133): built native rather than adopting the upstream loop -- that orchestrator couples phases in-process, targets a diverged phase set (params, which we dropped as null; no code phase), and bundles unrelated changes (a semantic-preservation constraint, a model-alias map, a judge cache, length penalties).
jramos
force-pushed
the
feat/cross-phase-orchestrator
branch
from
June 28, 2026 23:10
5ffa59e to
ed960d7
Compare
Address review findings on the cross-phase orchestrator:
- Stale-gate silent failure: per-phase --output-dir was deterministic and
never cleared, so a non-resume rerun where a phase died before rewriting its
gate would read the prior run's gate_decision.json and be recorded as
passed/deploy. Clear the phase run dir before each launch so a captured gate
always belongs to the current run.
- summary "deployable" now requires status==passed, not just decision==deploy,
so an aborted phase with a stale/partial deploy gate can't reach the
human-deploy list.
- --resume now treats only {passed, denied} as done: a crashed run resumes by
retrying its failed/aborted phase, and a dry-run row no longer masks real
execution.
- Add --phase-timeout: a wedged phase is killed (exit 124 -> failed) instead of
hanging the whole serial run.
- Collapse the redundant supports_create_pr field into a property derived from
create_pr_style, and raise on an unknown style instead of silently emitting no
PR flags.
- Guard load_done against a truncated ledger line (don't let a crash mid-append
break --resume); reject create_pr=null spend ceilings while still allowing 0;
reject output_dir inside a phase's args; log non-passed phases as they happen.
jramos
added a commit
that referenced
this pull request
Jul 6, 2026
…ne new item (#150) Incremental upstream review. Backlog steady at 72 open; six new PRs #142-#147. Five already covered -> SKIP; one genuinely-new latent bug -> new action item. - NousResearch#142: 5 of 6 fixes are covered clusters (GEPA/DSPy-3.2 compat x3, validate-full). The 6th is real and new here: find_skill via Path.rglob('SKILL.md') skips symlinked skill dirs on Python <3.13 (skill_sources.py:63/67/80). Promoted to a CHERRY-PICK action item (recommended). - #143: our #102 — state.db read-only importer already shipped; only cross-platform discovery extras. SKIP. - #144/#145: Sunwo0u HSE sanitized evidence packets, report-only, no mechanism. SKIP. - #146: extraction cluster — our skill_text is a @Property over signature.instructions, so its 'dead input field' premise is upstream-only. SKIP. - #147: compat + material_diff reporting subsumed by our behavioral deploy gate. SKIP. Review-log entry + new action-item row + 2026-07-06 snapshot delta.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sequences the per-subsystem evolvers (skills → tools → prompts → code) from one YAML run-spec instead of invoking each by hand:
Each phase runs as an isolated subprocess (fault containment, and it handles the CLI-only
codephase uniformly). The orchestrator captures each phase's existinggate_decision.jsonat a deterministic--output-dirand writes a JSONL run history +summary.json/summary.mdto its run root.New package
evolution/orchestrator/(mirrorsevolution/monitor/):spec.py— YAML run-spec loader/validator (SpecErrorat load, before any launch).adapters.py— one adapter per phase: argv builder + deterministic output dir + required fields + create-pr/cost-flag spelling.run.py— the sequencer with an injectedphase_runnerseam; gate-file-firstreconcile.history.py— append-onlyrun_history.jsonl+ summary rendering (thedeployablelist is the human-in-loop handoff).__main__.py— the click CLI.Propose-only by construction
create_prdefaults off; the CLI strips every per-phasecreate_prunless--allow-pris passed, and a surviving PR phase must carry its spend ceiling (mirrors the monitor's--attempt-topguard). The orchestrator has no deploy/PR code path of its own.status=passed, decision=reject— not a failure. Only{failed, aborted}halt under--stop-on-error(default is continue-on-error; phases are independent).--resumeskips phases already in the ledger;--onlyfilters;--dry-runrecords the resolved argv without spawning.Evolver change
Adds the missing
--output-dirCLI option to the skills and tools evolvers (theirevolve()already acceptedoutput_dir; only the CLI wiring was absent) so the orchestrator captures every phase's run dir uniformly. Prompts and code already exposed it.Why native (resolves our triage item for NousResearch#133)
The upstream PR is a real cross-phase orchestrator, but it couples phases in-process, targets a phase set this fork has diverged from (it has
params— dropped here as a null axis — and lacks ourcodephase), and bundles unrelated changes (a Jaccard semantic-preservation constraint, a model-alias map, a judge cache, length penalties). This implementation borrows the shape (JSONL history, fault isolation, dry-run/phase-subset CLI) and rejects the in-process coupling, the bundled cruft, and the cross-run auto-scheduler (which would cross the propose-only boundary).docs/upstream_pr_triage.mdis updated in this PR (row + review log + snapshot).Test plan
uv run pytest tests/orchestrator/ -q— 31 passed (spec/adapters/history/run/cli, fake-phase_runnerseam, no LM spend)uv run pytest tests/skills/test_evolve_skill_output_dir.py tests/tools/test_evolve_tool_output_dir.py -q— 4 passeduv run pytest tests/ -q -m 'not slow'— 1746 passed, 4 deselected--dry-runagainst the sample spec resolves correct, runnable argv for all four phases (every flag validated against each phase CLI's accepted options)