docs(triage): skill broad-benchmark gate already covered (NousResearch/hermes-agent-self-evolution#127) - #146
Merged
Merged
Conversation
…h#127) The broad-benchmark-regression deploy gate already exists on the skill path as the `--benchmark-cmd` hook (the shared `run_benchmark_hook` in evolution/core/quality_gate.py), symmetric with the code evolver's full-suite deploy tier. The framing in the triage row — "we have the regression-floor/oracle analogue for code only" — does not hold: the hook stages baseline/evolved, passes $EVOLVED_PATH/$BASELINE_PATH, fails closed on nonzero exit, and records the outcome in gate_decision.json on both the skill and code paths. Upstream NousResearch#127's TBLite gate (hard-coded command -> JSON {score|pass_rate} -> 2% absolute floor) is a strict special case of our BYO hook: the user runs the broad benchmark on both arms and applies any floor inside the command. The one thing the code path uniquely has — the automatic per-candidate tests/tools regression floor (evolution/code/gate.py) — has no cheap analogue for skills, which are scoped, per-task-isolated instruction artifacts that cannot deterministically break a sibling. Its only broad-regression failure mode (a widened trigger that mis-fires on unrelated tasks) is detectable solely by expensive agentic rollout — exactly what the opt-in hook provides. No code change. Added a skill broad-regression `--benchmark-cmd` recipe to docs/usage.md to retire the perception gap, and recorded the disposition in docs/upstream_pr_triage.md (review log + row + snapshot).
jramos
enabled auto-merge (squash)
June 29, 2026 02:16
jramos
added a commit
that referenced
this pull request
Jul 6, 2026
…ne new item (#150) Incremental upstream review. Backlog steady at 72 open; six new PRs #142-#147. Five already covered -> SKIP; one genuinely-new latent bug -> new action item. - NousResearch#142: 5 of 6 fixes are covered clusters (GEPA/DSPy-3.2 compat x3, validate-full). The 6th is real and new here: find_skill via Path.rglob('SKILL.md') skips symlinked skill dirs on Python <3.13 (skill_sources.py:63/67/80). Promoted to a CHERRY-PICK action item (recommended). - #143: our #102 — state.db read-only importer already shipped; only cross-platform discovery extras. SKIP. - #144/#145: Sunwo0u HSE sanitized evidence packets, report-only, no mechanism. SKIP. - #146: extraction cluster — our skill_text is a @Property over signature.instructions, so its 'dead input field' premise is upstream-only. SKIP. - #147: compat + material_diff reporting subsumed by our behavioral deploy gate. SKIP. Review-log entry + new action-item row + 2026-07-06 snapshot delta.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resolves the triage action item for upstream NousResearch#127 ("broad-benchmark-regression-as-a-gate, applied to the skill path"). Disposition: Investigated → already covered. No code change.
Finding
The broad-benchmark-regression deploy gate already exists on the skill path as the
--benchmark-cmdhook, and it is symmetric across evolvers:evolve_skill.pystagesevolved_skill.md/baseline_skill.md, passes$EVOLVED_PATH/$BASELINE_PATH/$RUN_DIR, fails closed on nonzero exit, and records the outcome ingate_decision.json— via the samerun_benchmark_hook(evolution/core/quality_gate.py) the code evolver uses for its full-suite deploy tier (evolve_code.py).benchmark_gate.py(hard-coded TBLite command → JSON{score|pass_rate}→ 2% absolute floor) is a strict special case of our BYO hook: the user runs the broad benchmark on both arms and applies any floor inside the command (a 2% floor reproduces upstream's threshold).So the triage premise — "we have the regression-floor/oracle analogue for code only" — does not hold.
The one thing the code path uniquely has — the automatic per-candidate
tests/toolsregression floor (evolution/code/gate.py, baseline-vs-repaired diff) — has no cheap analogue for skills. A skill is a scoped, opt-in instruction artifact installed in isolation per task (SkillFileInstaller); it cannot deterministically break a sibling the way a code edit to tool A breaks tool B's unit test. Its only broad-regression failure mode (a widened trigger that mis-fires on unrelated tasks) is stochastic agentic behavior detectable solely by expensive rollout — exactly what the opt-in hook provides — so no automatic skill backstop is warranted.(Note: the standalone
evolution.validation.closed_loopCLI is not the skill's broad gate — it is tool-only, requires--tool/--hermes-repo, and even inline scores the skill's own task suite, not a broad benchmark.)Changes (doc-only)
docs/usage.md— added a skill broad-regression--benchmark-cmdrecipe (run the broad benchmark on both arms, exit nonzero past a floor) so the mechanism is demonstrated concretely, not merely asserted.docs/upstream_pr_triage.md— review-log entry + flipped row feat(evolve_skill): --closed-loop-gate-primary — gate on the behavioral oracle #127 to ✅ with the disposition + snapshot update.Verification
Doc-only; the disposition was cross-checked by an adversarial review pass that confirmed the conclusion rests solely on
--benchmark-cmd(and corrected an earlier draft that leaned on the tool-onlyclosed_loop.py).