diff --git a/docs/concepts/ooda-stuck-label-self-heal.md b/docs/concepts/ooda-stuck-label-self-heal.md new file mode 100644 index 000000000..3a416e2cb --- /dev/null +++ b/docs/concepts/ooda-stuck-label-self-heal.md @@ -0,0 +1,192 @@ +--- +title: The `ooda-stuck` escalation self-heals the missing label instead of failing silently +description: Why a blocked OODA goal can always be escalated to the operator through the GitHub-issue channel — even when the `ooda-stuck` label does not exist in the target repo. The three breaker/safeguard filers now idempotently ensure the label (create-if-missing) before filing, and gracefully degrade to filing the issue WITHOUT the label (with a structured WARN) when the label cannot be created, so `gh issue create` never exits non-zero on an unknown label and the escalation always succeeds. Preserves one-issue-per-stall idempotency; additive and non-breaking (issue #4474). +last_updated: 2026-07-22 +review_schedule: as-needed +owner: simard +doc_type: concept +status: implemented +related: + - ../reference/ooda-stuck-label-self-heal-api.md + - ../reference/no-progress-breaker-api.md + - ../reference/no-progress-root-cause-resolution-api.md + - ./no-progress-root-cause-resolution.md + - ./blocked-goal-escalation-backoff.md + - ./steerable-ooda-daemon.md + - ../howto/diagnose-a-no-progress-block.md + - ../reference/goal-labels.md + - ../../src/stewardship/gh_client.rs + - ../../src/ooda_loop/no_progress.rs + - ../../src/ooda_actions/advance_goal/spawn.rs +--- + +# The `ooda-stuck` escalation self-heals the missing label + +> **Status: implemented (issue #4474).** The three sites that file an +> `ooda-stuck` tracking issue — the no-progress breaker production filer, the +> brain-failure deterministic safeguard, and the engineer-lifecycle +> open-tracking-issue path — now route their label handling through a single +> shared helper, +> [`ensure_label`](https://github.com/rysweet/Simard/blob/main/src/stewardship/gh_client.rs), +> in `src/stewardship/gh_client.rs`. The helper idempotently creates the label +> if it is missing and, when it cannot, tells the caller to file the issue +> **without** the label rather than let `gh` fail. Primary sources: +> [`src/stewardship/gh_client.rs`](https://github.com/rysweet/Simard/blob/main/src/stewardship/gh_client.rs) +> (`ensure_label`, `LabelDisposition`, `LabelEnsureExecutor`), +> [`src/ooda_loop/no_progress.rs`](https://github.com/rysweet/Simard/blob/main/src/ooda_loop/no_progress.rs) +> (`GhIssueFiler`), and +> [`src/ooda_actions/advance_goal/spawn.rs`](https://github.com/rysweet/Simard/blob/main/src/ooda_actions/advance_goal/spawn.rs) +> (the deterministic safeguard and open-tracking-issue sites). API details: +> [`ooda-stuck` label self-heal reference](../reference/ooda-stuck-label-self-heal-api.md). + +## The defect this fixes + +When the OODA no-progress breaker fires for a goal it cannot un-stick, its last +resort is to escalate to a human by filing a GitHub tracking issue. Every filer +tagged that issue with the `ooda-stuck` label: + +```bash +gh issue create --title "…" --body "…" --label ooda-stuck +``` + +But the `ooda-stuck` label **did not exist** in `rysweet/Simard`. `gh issue +create` treats an unknown `--label` as a hard error and exits non-zero: + +``` +could not add label: ooda-stuck not found +``` + +Because the filer only ever checked the exit status, a non-zero exit meant the +issue was **never created** — the escalation silently failed. The breaker had +already marked the goal `Blocked` with its sentinel, but no issue reached the +operator, and the goal stayed parked with no visible escalation artifact. + +The journal signature (`simard::ooda`, `ERROR`) recorded the fingerprint: + +``` +no-progress breaker: gh issue create failed (goal still Blocked) + stderr=could not add label: ooda-stuck not found +``` + +This recurred every ~45 minutes as the breaker re-fired across OODA cycles +2430 / 2433 / 2436 / 2439 (five occurrences, 15:47 → 21:10). The blocked-goal +escalation channel through GitHub issues was, in practice, dead. + +Three call sites shared the defect, each with its own subtlety: + +| Site | File | Prior behaviour | +| --- | --- | --- | +| No-progress breaker production filer | `src/ooda_loop/no_progress.rs` (`GhIssueFiler::file_issue`, ~line 116) | `--label ooda-stuck` → non-zero exit → `ERROR` log, returned `None`, goal stayed `Blocked` with no linked issue. | +| Brain-failure deterministic safeguard | `src/ooda_actions/advance_goal/spawn.rs` (~lines 378–379) | `--label ooda-stuck` → non-zero exit → `ERROR` log; the safeguard-`Blocked` mark survived but no issue was filed. | +| Engineer-lifecycle open-tracking-issue | `src/ooda_actions/advance_goal/spawn.rs` (~lines 935–936) | `--label ooda-stuck` **and** used `.status()`, which drops captured streams — a non-zero exit produced **no** log at all (a second, latent silent-failure bug). | + +## The fix: ensure-or-degrade, never fail on a missing label + +The fix is a single, shared label-handling seam that both **self-heals** the +missing label and **fail-safe degrades** when it cannot — so the escalation +issue is filed either way. + +### 1. Ensure the label idempotently (create-if-missing) + +Before filing, each site calls `ensure_label("ooda-stuck")`. The helper runs: + +```bash +gh label create ooda-stuck +``` + +> **Design decision — repo scoping (ambient, not `-R`).** `ensure_label` uses +> the same ambient repository context (current working directory) as the +> sibling `gh issue create` call at each site — it deliberately does **not** +> pass `-R `. This is both simpler and *more correct*: the original +> failure signature was `could not add label: ooda-stuck not found`, which +> proves `gh issue create` had already resolved the repo from the ambient cwd +> and failed *only* on the label. Creating the label in that same ambient repo +> guarantees the label and the issue always target the identical repository. An +> earlier draft of the design passed an explicit `-R ` slug to +> `gh label create`; that was rejected because (a) none of the three call sites +> has a validated `owner/repo` slug in scope — the deterministic safeguard runs +> *before* `goal_repo_slug` is resolved, and neither `apply_lifecycle_decision` +> nor `GhIssueFiler` receives a repo — and (b) resolving a slug independently +> risks creating the label in a *different* repo than `gh issue create` targets. +> Keeping both commands ambient sidesteps both problems and keeps the change +> additive (no repo threading, no trait/struct signature change). + +- **Exit 0** → the label now exists → attach it. +- **"already exists"** (case-insensitive stderr match) → the label was already + there → attach it. This is what makes the call idempotent: a label created on + the first stall is simply re-observed on the next, never re-created, never an + error. + +This is the create-if-missing self-heal: the first time any goal stalls in a +repo without the label, Simard creates the label once and every subsequent +escalation reuses it. + +### 2. Degrade gracefully when the label can't be created + +A token that can *file issues* may not have permission to *create labels* +(label creation needs repo-write). If `gh label create` fails for any +reason other than "already exists" — an authorization error, a spawn failure — +the helper does **not** propagate an error and does **not** abort the +escalation. It returns a `LabelDisposition::Omit { reason }`, and the caller +files the issue **without** the `--label` argument: + +```bash +gh issue create --title "…" --body "…" # no --label; escalation still succeeds +``` + +The degradation is always surfaced with a structured `tracing::warn` on the +site's own target (`simard::ooda` or `simard::ooda_brain`), carrying the reason. +This honours the **no-silent-fallback** rule: the issue is still filed, but the +operator can see in the logs that the label was omitted and why. + +### 3. Stop swallowing exit codes at the open-tracking-issue site + +The engineer-lifecycle site (`spawn.rs` ~line 935) previously used `.status()`, +which discards captured stdout/stderr and — combined with only logging the +`Err(_)` spawn case — meant a non-zero `gh` exit produced no diagnostics at all. +It now uses `.output()` and inspects the exit status, emitting a structured +`warn` with the lossy-decoded, length-bounded (≤ 2 KiB) stderr on failure. This +closes the latent second bug so that site can never silently fail again either. + +## What is preserved + +- **One issue per stall (idempotency).** The self-heal changes only *label* + handling. The dedup that guarantees a stalled goal is escalated exactly once — + the breaker-authored `[no-progress-tracking]` `WipRef` link and the + `already_tracked` check in the escalation path — is untouched. A re-stall + never spams duplicate `ooda-stuck` issues. See + [the no-progress breaker API](../reference/no-progress-breaker-api.md) and + [blocked-goal escalation backoff](./blocked-goal-escalation-backoff.md). +- **Existing tracing targets and return contracts.** Each caller keeps its own + target (`simard::ooda` vs `simard::ooda_brain`) and return type + (`Option` vs unit). Only the label concern is centralised. +- **The `eprintln!` operator line** at the deterministic-safeguard success path + (`spawn.rs` ~line 384) is unchanged; no new `print!`/`println!` is introduced. +- **argv-only, no-shell invocation.** The label is a hardcoded constant (no + leading `-`) and no repo argument is added, so the new `gh label create` + subprocess call is as injection-safe as the existing ones — there is no + attacker-influenced argv value at all. + +## End-to-end effect + +A blocked goal is now **always** escalable through the GitHub-issue channel: + +1. Breaker fires, marks the goal `Blocked` with its sentinel. +2. `ensure_label` creates `ooda-stuck` on the first stall (or observes it, or + degrades to no-label with a `WARN`). +3. `gh issue create` succeeds — with the label when possible, without it when + not — and the operator receives the tracking issue. +4. The filed issue is linked back to the goal as its + `[no-progress-tracking]` artifact, and the dedup guard prevents any duplicate + on a re-stall. + +The `could not add label: ooda-stuck not found` error class can no longer break +escalation. + +## See also + +- [`ooda-stuck` label self-heal API reference](../reference/ooda-stuck-label-self-heal-api.md) +- [No-progress breaker API reference](../reference/no-progress-breaker-api.md) +- [Concept: the no-progress breaker explains WHY and self-resolves before escalating](./no-progress-root-cause-resolution.md) +- [How to diagnose a no-progress block](../howto/diagnose-a-no-progress-block.md) +- [Goal labels / tags API reference](../reference/goal-labels.md) diff --git a/docs/howto/diagnose-a-no-progress-block.md b/docs/howto/diagnose-a-no-progress-block.md index 27952547b..75b5a7af9 100644 --- a/docs/howto/diagnose-a-no-progress-block.md +++ b/docs/howto/diagnose-a-no-progress-block.md @@ -79,6 +79,15 @@ The matching escalation issue (label `ooda-stuck`) carries the same WHY plus the agentic narrative paragraph, if the [optional WHY recipe](../reference/ooda-no-progress-why-recipe.md) is enabled. +> **The `ooda-stuck` label self-heals.** If the label is missing from the target +> repo, the escalation no longer fails: the filers idempotently create it +> (create-if-missing) or file the issue **without** the label (with a structured +> `WARN`) rather than let `gh issue create` exit non-zero. So a stalled goal is +> always escalable through the issue channel — an escalation issue **without** the +> `ooda-stuck` label just means the label could not be created (check the `WARN`). +> See [`ooda-stuck` label self-heal](../concepts/ooda-stuck-label-self-heal.md) +> and its [API reference](../reference/ooda-stuck-label-self-heal-api.md). + > If you see a block **without** a `why=` segment, it is either a legacy block > from before this upgrade or a block authored by a *different* path (operator > hold, scope, dependency, or the brain-failure safeguard). Those are out of diff --git a/docs/index.md b/docs/index.md index fbce969e7..35463aa44 100644 --- a/docs/index.md +++ b/docs/index.md @@ -57,6 +57,7 @@ Terminal sessions and repo-grounded engineer runs now bridge through one explici - [Concept: standing/perpetual goals are exempt from the no-progress hard-block](./concepts/perpetual-goal-no-progress-exemption.md) — why a bursty standing goal (the continuous self-research goal) must never be parked "needs human review" by the OODA no-progress safeguard; the runtime exemption plus the load-time self-heal that keep it continuous and self-sustaining without operator unblocking (#2589). See the [no-progress breaker API](./reference/no-progress-breaker-api.md) and the [unblock runbook](./howto/unblock-stuck-ooda-goals.md). - [Concept: the standing research goal never idles — an idle cycle is a fault](./concepts/research-goal-never-idle.md) — why Simard's standing cognition-research goal must produce a concrete NOVEL action **every** cycle (a new external source ingestion OR a new measurable experiment, dedup'd against recent directions) and why an idle cycle for THIS goal is a **fault** the daemon re-orients out of — not the benign perpetual-idle exemption other standing goals keep. Prompt-first (charter + never-idle directive), reinforced by a thin fail-closed breaker rail (`classify_standing_idle` → `research_idle_faults` → re-orient, never block) keyed on `is_standing_research_goal()` (#4399). See the [never-idle rail API](./reference/research-goal-never-idle-rail-api.md) and the [keep-the-research-goal-never-idle how-to](./howto/keep-the-research-goal-never-idle.md). - [Concept: the no-progress breaker explains WHY and self-resolves before escalating](./concepts/no-progress-root-cause-resolution.md) — why the safeguard no longer parks a goal with a bare "needs human review": it classifies the root cause of a stall (already-complete / obsolete / missing-precondition / upstream-dependency / unclear-criteria / genuinely-stuck), self-resolves the machine-fixable causes (auto-complete on live artifacts — the fix for the `kgpacks-rs` "already done" goals — clone a missing repo, defer behind an upstream, spawn one guided engineer), and escalates to a human only as a last resort with the concrete WHY + evidence attached. See the [root-cause resolution API](./reference/no-progress-root-cause-resolution-api.md), the [`ooda-no-progress-why` recipe](./reference/ooda-no-progress-why-recipe.md), and the [diagnose-a-no-progress-block runbook](./howto/diagnose-a-no-progress-block.md). +- [Concept: the `ooda-stuck` escalation self-heals the missing label](./concepts/ooda-stuck-label-self-heal.md) — why a blocked OODA goal can always be escalated to the operator through the GitHub-issue channel even when the `ooda-stuck` label does not exist in the target repo: the three breaker/safeguard filers (no-progress production filer, brain-failure deterministic safeguard, engineer-lifecycle open-tracking-issue) now route their `--label ooda-stuck` handling through the shared `ensure_label` helper, which idempotently creates the label (create-if-missing) or degrades to filing the issue WITHOUT the label plus a structured `WARN` when it cannot — so `gh issue create` never exits non-zero on the unknown label and the escalation always succeeds, while one-issue-per-stall idempotency is preserved (#4474). See the [label self-heal API reference](./reference/ooda-stuck-label-self-heal-api.md) and the [diagnose-a-no-progress-block runbook](./howto/diagnose-a-no-progress-block.md). - [Concept: the terminal no-progress stall never parks a goal with empty evidence](./concepts/no-progress-terminal-investigation.md) — the issue #16 fix for the 12–13 of 20 live goals stamped `why=GENUINELY-STUCK evidence=[(none)]`: the terminal rung reuses the existing guided-engineer (independent recipe-runner) investigation rather than inventing a parallel path, escalates only WITH concrete evidence, and otherwise surfaces a fail-visible investigation gap (`SurfaceInvestigationFailure` → `investigation_errors`) instead of a bare block. Evidence-less `(none)` blocks are folded into the re-investigation population (`needs_reinvestigation`), and a per-signature overseer inflight guard (`inflight_investigations`) prevents duplicate concurrent investigations of the same `overseer-obs:goal:blocked:…` signature. - [Concept: brain-relative OODA cycle counter](./concepts/brain-relative-ooda-cycle-counter.md) — why "Cycle #N" must count the **brain's total lived cognition** (a durable, monotonic number that continues across every daemon restart and deploy) instead of the current process's uptime, which reset the dashboard to "Cycle #1" on every deploy and erased the visible sense of accumulated thought — persisted in the same durable `goal_board.json` store as the goal board and the no-progress tracker. See the [durable cycle counter API](./reference/durable-ooda-cycle-counter.md) and the [inspect-and-verify how-to](./howto/inspect-the-ooda-cycle-counter.md). - [Concept: Overseer goal-board health](./concepts/overseer-goal-board-health.md) — the steward-side defense-in-depth complement to #2609: the acting Overseer now observes `Blocked` goals on the board, **self-heals** a false-parked perpetual goal (auto-unblock + reactivate, the exact `simard goal unblock`), and **escalates** any "needs human review" block to the operator on both channels — closing the silent-failure gap where a parked standing goal reached no human (#2616). See the [goal-board health API](./reference/overseer-goal-board-health-api.md) and the [configure-and-observe how-to](./howto/configure-overseer-goal-board-health.md). @@ -218,6 +219,7 @@ If you are changing architecture, start with the [architecture overview](./archi - [Research-goal never-idle rail API reference](./reference/research-goal-never-idle-rail-api.md) — the #4399 never-idle rail on top of the breaker: the shared `classify_standing_idle` classifier consumed by both breaker sites, the `StandingIdle` split (benign-exempt vs research-fault), the fixed `ResearchIdleFault` vocabulary, the additive `NoProgressBreakerReport.research_idle_faults` field (excluded from `fired()`), the `roll_to_new_cycle` re-orient, and the revised never-idle directive contract (NEW source OR NEW experiment, dedup'd, local-experiment floor — no idle fallback). See the [concept](./concepts/research-goal-never-idle.md) and the [keep-the-research-goal-never-idle how-to](./howto/keep-the-research-goal-never-idle.md). - [wip-ref liveness reconcile API reference](./reference/wip-ref-liveness-reconcile-api.md) — the #4428 two-prong per-cycle reconcile (dead-session sweep + merged/closed-PR prune) that makes `has_live_in_flight_ref()` reflect true liveness before the never-idle breaker classifies, plus the [round-3 hardening reference](./reference/wip-ref-liveness-reconcile-hardening-api.md) that makes both prongs total: dead-ref prune for **every** goal regardless of assignment (FIX-1) and a merged-PR reconcile scoped to **each goal's own repo** instead of a hardcoded `rysweet/Simard` (FIX-2). - [No-progress root-cause resolution API reference](./reference/no-progress-root-cause-resolution-api.md) — the root-cause layer on the breaker: the `NoProgressClass` classification and its stable tokens, the `NoProgressWhy`/`Evidence` types, the optional `NoProgressWhyReasoner` agentic-enrichment trait, the extended `NoProgressResolution` ladder (auto-complete / heal / defer / one-shot guided-retry / escalate-with-WHY), the WHY-bearing block-reason contract, the additive `EvidenceSource::repo_present` and `dependency_goal_state` methods, and the fail-closed error table. See the [concept](./concepts/no-progress-root-cause-resolution.md) and the [diagnose runbook](./howto/diagnose-a-no-progress-block.md). +- [`ooda-stuck` label self-heal API reference](./reference/ooda-stuck-label-self-heal-api.md) — the shared label helper the escalation filers use so a missing `ooda-stuck` label can never break `gh issue create`: the `ensure_label` free function, the `LabelDisposition` (`Attach` / `Omit { reason }`) contract, the `LabelEnsureExecutor` fn-pointer test seam (mirroring `CreateIssueExecutor`), the create-if-missing / already-exists idempotency, the fail-safe degrade-to-no-label path, and how the three call sites (`no_progress.rs` `GhIssueFiler`, the `spawn.rs` deterministic safeguard, and the `spawn.rs` open-tracking-issue site — which also switches `.status()` → `.output()` to stop swallowing exit codes) gate their `--label ooda-stuck` argument on the disposition (#4474). See the [concept](./concepts/ooda-stuck-label-self-heal.md). - [The `ooda-no-progress-why` recipe reference](./reference/ooda-no-progress-why-recipe.md) — the optional agentic-enrichment recipe that turns the deterministic classification + evidence into a human-readable WHY narrative for the escalation: context variables, the single-string narrative output contract, the "enrichment never changes routing" invariant, hot-reload resolution order, and the NO-FALLBACK fail-closed contract. - [Durable OODA cycle counter API reference](./reference/durable-ooda-cycle-counter.md) — the brain-relative, monotonic cycle counter: the persisted `PersistentGoalState.cycle_count` field, the `commit_cycle` `max()` guard, the daemon startup seed, the one-time cycle-report backfill, the repointed `daemon_health.json` writes, and the single source of truth that keeps every `cycle=` log, cycle report, and dashboard "Cycle #N" counting the brain's total lived cognition instead of resetting to 1 on each restart. See the [concept](./concepts/brain-relative-ooda-cycle-counter.md) and the [inspect-and-verify how-to](./howto/inspect-the-ooda-cycle-counter.md). - [Goal decomposition & the goal graph](./reference/goal-decomposition.md) — break a large goal into 2–6 bounded sub-goals and record parent↔child structure as typed, queryable edges in the cognitive-memory graph, with parent-progress roll-up and the `simard goal decompose` verb (#2405). diff --git a/docs/reference/no-progress-breaker-api.md b/docs/reference/no-progress-breaker-api.md index 3cacf870e..883bb7dd2 100644 --- a/docs/reference/no-progress-breaker-api.md +++ b/docs/reference/no-progress-breaker-api.md @@ -9,6 +9,8 @@ status: implemented related: - ../concepts/perpetual-goal-no-progress-exemption.md - ../concepts/no-progress-root-cause-resolution.md + - ../concepts/ooda-stuck-label-self-heal.md + - ../reference/ooda-stuck-label-self-heal-api.md - ../concepts/steerable-ooda-daemon.md - ../reference/no-progress-root-cause-resolution-api.md - ../reference/ooda-no-progress-why-recipe.md diff --git a/docs/reference/ooda-stuck-label-self-heal-api.md b/docs/reference/ooda-stuck-label-self-heal-api.md new file mode 100644 index 000000000..bbf2e6d5b --- /dev/null +++ b/docs/reference/ooda-stuck-label-self-heal-api.md @@ -0,0 +1,239 @@ +--- +title: "`ooda-stuck` label self-heal API reference" +description: Reference for the shared label self-heal used by the OODA no-progress breaker and brain-failure safeguard filers — the `ensure_label` free function, the `LabelDisposition` (Attach / Omit) contract, the `LabelEnsureExecutor` fn-pointer test seam, and how the three `gh issue create` call sites gate their `--label ooda-stuck` argument on the disposition so a missing label self-heals (create-if-missing) or degrades (file without label + WARN) instead of failing the escalation (issue #4474). +last_updated: 2026-07-22 +review_schedule: as-needed +owner: simard +doc_type: reference +status: implemented +related: + - ../concepts/ooda-stuck-label-self-heal.md + - ./no-progress-breaker-api.md + - ./no-progress-root-cause-resolution-api.md + - ./goal-labels.md + - ./stewardship-api.md + - ../howto/diagnose-a-no-progress-block.md + - ../../src/stewardship/gh_client.rs + - ../../src/ooda_loop/no_progress.rs + - ../../src/ooda_actions/advance_goal/spawn.rs +--- + +# `ooda-stuck` label self-heal API reference + +> **Status: implemented (issue #4474).** The label self-heal helper lives in +> [`src/stewardship/gh_client.rs`](https://github.com/rysweet/Simard/blob/main/src/stewardship/gh_client.rs). +> The three consuming filers are the no-progress breaker's `GhIssueFiler` in +> [`src/ooda_loop/no_progress.rs`](https://github.com/rysweet/Simard/blob/main/src/ooda_loop/no_progress.rs) +> and the deterministic-safeguard + open-tracking-issue sites in +> [`src/ooda_actions/advance_goal/spawn.rs`](https://github.com/rysweet/Simard/blob/main/src/ooda_actions/advance_goal/spawn.rs). +> Narrative: [The `ooda-stuck` escalation self-heals the missing label](../concepts/ooda-stuck-label-self-heal.md). + +The escalation filers used to pass `--label ooda-stuck` unconditionally. If the +label did not exist in the target repo, `gh issue create` exited non-zero +(`could not add label: ooda-stuck not found`) and the issue was never filed — +the escalation failed silently. This reference documents the additive helper +that makes the label handling **ensure-or-degrade**, so the issue is always +filed. + +## Overview + +``` +ensure_label(label) ──► LabelDisposition + ├─ Attach (label exists / was created) + └─ Omit { reason } (label un-ensurable — file without it) +``` + +Each caller invokes `ensure_label` before building its `gh issue create` argv, +then conditionally appends `--label