0.5 — Wire make kb-eval as CI regression gate

[bb-connor]

Phase 0 task 0.5 (PLAN.md, Phase 0 task 0.5).

Wire make kb-eval as a CI gate that fails on retrieval regression OR outcome-eval regression.

Deliverables

• GitHub Actions workflow .github/workflows/eval.yml:
  • Runs make kb-eval-outcomes on every PR.
  • Once Phase 1 lands, also runs make kb-eval-retrieval (gated on stack-up).
  • Posts the vault/_meta/dashboards/eval-outcomes.md table as a PR comment.
• CI red on:
  • Retrieval eval below overall A on any of the 9 PR #599 categories.
  • Any outcome eval beyond its regression_threshold from rolling 30-day mean (per outcomes.yml).
  • Eval 4 disagreement-flag rate > 30%.
• Negative test: a make test-regression target that intentionally regresses one eval and confirms make kb-eval exits non-zero.

Done when

• A PR that intentionally introduces a regression turns the eval check red.
• A PR with no regression passes cleanly.

Reference

• PHASE-0.md "Regression policy"
• PHASE-0.md "Phase 0 done"
• outcomes.yml regression_threshold

[bb-connor]

Partial flip landed in 3732954

The workflow now has three blocking gates (none with continue-on-error). Verified end-to-end on the push run 25529563543 — all green.

Blocking gates today

Step	Catches
Engine ↔ pack boundary check (ops/ci/check-imports.py)	Anyone introducing a chio_* import in kb-engine/
kb-engine unit tests (8 tests)	Plugin protocol break, Registry dispatch regression, dataclass equality break
chio-pack unit tests (26 tests)	Fixture schema regression, AST-level boundary regression, plugin registration break, heuristic mistake classifier break, session-log JSONL contract break

Still informational (deliberately)

The "Run outcome evals" step stays continue-on-error: true because the runners return blocked-input until ADR-0002 baselines exist. The four runners need:

• time-to-first-correct-fix — Phase 1.5 agent-runner spawning
• repeated-mistake-rate — 20+ logged chio-dev sessions (Phase 1.5 CLI)
• conformance-harness-recall — Phase 1.x KB stack with kb_search_*
• capability-error-explanation — Phase 1.x rater pipeline (ADR-0004 raters wired)

Final flip criterion

Per the workflow header comment:

When status: ok across all four evals, flip continue-on-error: false — that's the canonical close-out of issue #5.

This issue stays open for the final flip. The blocking surface today is genuinely useful (catches every regression we can describe in code), but the named goal of issue #5 — "the gate becomes real for outcome evals" — depends on Phase 1.x deliverables landing first.

Related

• ADR-0004 (58c24ae) unblocks the rater pipeline.
• Phase 1.2 kb-engine + chio-pack plugin (57f6624) makes the boundary CI-enforceable.
• Phase 1.1 docker stack (dea5872) makes make kb-up reachable.