Skip to content

Add a native cross-phase evolution orchestrator - #145

Merged
jramos merged 2 commits into
mainfrom
feat/cross-phase-orchestrator
Jun 29, 2026
Merged

Add a native cross-phase evolution orchestrator#145
jramos merged 2 commits into
mainfrom
feat/cross-phase-orchestrator

Conversation

@jramos

@jramos jramos commented Jun 28, 2026

Copy link
Copy Markdown
Owner

Summary

Sequences the per-subsystem evolvers (skills → tools → prompts → code) from one YAML run-spec instead of invoking each by hand:

python -m evolution.orchestrator --spec examples/orchestrator/sample_run.yaml --dry-run
python -m evolution.orchestrator --spec examples/orchestrator/sample_run.yaml --only skills --only tools

Each phase runs as an isolated subprocess (fault containment, and it handles the CLI-only code phase uniformly). The orchestrator captures each phase's existing gate_decision.json at a deterministic --output-dir and writes a JSONL run history + summary.json/summary.md to its run root.

New package evolution/orchestrator/ (mirrors evolution/monitor/):

  • spec.py — YAML run-spec loader/validator (SpecError at load, before any launch).
  • adapters.py — one adapter per phase: argv builder + deterministic output dir + required fields + create-pr/cost-flag spelling.
  • run.py — the sequencer with an injected phase_runner seam; gate-file-first reconcile.
  • history.py — append-only run_history.jsonl + summary rendering (the deployable list is the human-in-loop handoff).
  • __main__.py — the click CLI.

Propose-only by construction

  • create_pr defaults off; the CLI strips every per-phase create_pr unless --allow-pr is passed, and a surviving PR phase must carry its spend ceiling (mirrors the monitor's --attempt-top guard). The orchestrator has no deploy/PR code path of its own.
  • Verdict status is grounded in the gate file, never the evolvers' inconsistent exit codes (skills/tools exit 1 on setup errors before any gate; prompts exits 0 on dry-run). A clean run whose gate rejected the candidate is status=passed, decision=reject — not a failure. Only {failed, aborted} halt under --stop-on-error (default is continue-on-error; phases are independent).
  • --resume skips phases already in the ledger; --only filters; --dry-run records the resolved argv without spawning.

Evolver change

Adds the missing --output-dir CLI option to the skills and tools evolvers (their evolve() already accepted output_dir; only the CLI wiring was absent) so the orchestrator captures every phase's run dir uniformly. Prompts and code already exposed it.

Why native (resolves our triage item for NousResearch#133)

The upstream PR is a real cross-phase orchestrator, but it couples phases in-process, targets a phase set this fork has diverged from (it has params — dropped here as a null axis — and lacks our code phase), and bundles unrelated changes (a Jaccard semantic-preservation constraint, a model-alias map, a judge cache, length penalties). This implementation borrows the shape (JSONL history, fault isolation, dry-run/phase-subset CLI) and rejects the in-process coupling, the bundled cruft, and the cross-run auto-scheduler (which would cross the propose-only boundary). docs/upstream_pr_triage.md is updated in this PR (row + review log + snapshot).

Test plan

  • uv run pytest tests/orchestrator/ -q — 31 passed (spec/adapters/history/run/cli, fake-phase_runner seam, no LM spend)
  • uv run pytest tests/skills/test_evolve_skill_output_dir.py tests/tools/test_evolve_tool_output_dir.py -q — 4 passed
  • uv run pytest tests/ -q -m 'not slow' — 1746 passed, 4 deselected
  • End-to-end --dry-run against the sample spec resolves correct, runnable argv for all four phases (every flag validated against each phase CLI's accepted options)
  • pre-commit hygiene hooks pass

Sequence the per-subsystem evolvers (skills -> tools -> prompts -> code) from
one YAML run-spec instead of invoking each by hand. `python -m
evolution.orchestrator` runs each phase as an isolated subprocess, captures its
existing gate_decision.json verdict at a deterministic --output-dir, and writes
a JSONL run history + summary.

Propose-only by construction: it never deploys, and strips every per-phase
create_pr unless --allow-pr is passed (a surviving PR phase must carry its spend
ceiling). Verdict status is grounded in the gate file, not the evolvers'
inconsistent exit codes -- a clean run whose gate rejected the candidate is a
pass, not a failure. Fault isolation is the subprocess boundary;
continue-on-error by default, --stop-on-error to halt, --resume to skip phases
already in the ledger.

Also adds the missing --output-dir CLI option to the skills and tools evolvers
(their evolve() already accepted it) so the orchestrator captures every phase's
run dir uniformly.

Resolves our triage item for the upstream cross-phase orchestrator
(NousResearch#133): built native rather than
adopting the upstream loop -- that orchestrator couples phases in-process,
targets a diverged phase set (params, which we dropped as null; no code phase),
and bundles unrelated changes (a semantic-preservation constraint, a
model-alias map, a judge cache, length penalties).
@jramos
jramos force-pushed the feat/cross-phase-orchestrator branch from 5ffa59e to ed960d7 Compare June 28, 2026 23:10
Address review findings on the cross-phase orchestrator:

- Stale-gate silent failure: per-phase --output-dir was deterministic and
  never cleared, so a non-resume rerun where a phase died before rewriting its
  gate would read the prior run's gate_decision.json and be recorded as
  passed/deploy. Clear the phase run dir before each launch so a captured gate
  always belongs to the current run.
- summary "deployable" now requires status==passed, not just decision==deploy,
  so an aborted phase with a stale/partial deploy gate can't reach the
  human-deploy list.
- --resume now treats only {passed, denied} as done: a crashed run resumes by
  retrying its failed/aborted phase, and a dry-run row no longer masks real
  execution.
- Add --phase-timeout: a wedged phase is killed (exit 124 -> failed) instead of
  hanging the whole serial run.
- Collapse the redundant supports_create_pr field into a property derived from
  create_pr_style, and raise on an unknown style instead of silently emitting no
  PR flags.
- Guard load_done against a truncated ledger line (don't let a crash mid-append
  break --resume); reject create_pr=null spend ceilings while still allowing 0;
  reject output_dir inside a phase's args; log non-passed phases as they happen.
@jramos
jramos merged commit 10d96e6 into main Jun 29, 2026
4 checks passed
@jramos
jramos deleted the feat/cross-phase-orchestrator branch June 29, 2026 00:11
jramos added a commit that referenced this pull request Jul 6, 2026
…ne new item (#150)

Incremental upstream review. Backlog steady at 72 open; six new PRs #142-#147.
Five already covered -> SKIP; one genuinely-new latent bug -> new action item.

- NousResearch#142: 5 of 6 fixes are covered clusters
  (GEPA/DSPy-3.2 compat x3, validate-full). The 6th is real and new here: find_skill
  via Path.rglob('SKILL.md') skips symlinked skill dirs on Python <3.13
  (skill_sources.py:63/67/80). Promoted to a CHERRY-PICK action item (recommended).
- #143: our #102 — state.db read-only importer already shipped; only cross-platform
  discovery extras. SKIP.
- #144/#145: Sunwo0u HSE sanitized evidence packets, report-only, no mechanism. SKIP.
- #146: extraction cluster — our skill_text is a @Property over signature.instructions,
  so its 'dead input field' premise is upstream-only. SKIP.
- #147: compat + material_diff reporting subsumed by our behavioral deploy gate. SKIP.

Review-log entry + new action-item row + 2026-07-06 snapshot delta.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant