Skip to content

docs(operations): record CR-DD-016 Track B classifier-fallback investigation - #181

Merged
coreytshaffer merged 3 commits into
mainfrom
claude/track-b-classifier-fallback-investigation
Aug 29, 2026
Merged

docs(operations): record CR-DD-016 Track B classifier-fallback investigation#181
coreytshaffer merged 3 commits into
mainfrom
claude/track-b-classifier-fallback-investigation

Conversation

@coreytshaffer

Copy link
Copy Markdown
Owner

Summary

Read-only investigation record for Track B of CR-DD-016's Sequencing section, which required that track stay read-only "until its own findings warrant a separately scoped CR." This is that read-only output. One new file, 301 lines, documentation only.

No CR identifier is minted, no implementation authority is requested or granted, and no triage_core/ source, schema, test, or fixture is changed by this PR.

Track B was scoped to settle three questions. All three are now answered:

  1. Is "refactor" genuinely the universal terminal fallback? Yes. classify_deterministic's terminal no-match return (classifier.py:78) is reached by every path through classify (classifier.py:58) — no backend, construction failure, exception, timeout, empty response, or non-member response. Confirmed empirically: the empty string classifies as refactor.

  2. What downstream routing depends on it? This sharpens CR-DD-016's characterization. That CR called the fallback "semantically wrong"; the record establishes it is wrong in the permissive directionrefactor maps to sensitivity="low", routes to local_heavy, and yields human_review_required=False. An unclassifiable prompt fails toward autonomous execution rather than toward review. This is a property of the composition of classifier.py and client.py's maps; neither file states it.

  3. Did any trial traverse it? The Aug-8 trial motivating CR-DD-016 provably did not — confirming that CR's Corrective Amendment by probe rather than by inference. But the general question is unanswerable from recorded evidence by construction: bugfix and refactor both collapse onto task_class=code_repair before anything is written, so the two code_repair trials in 2026-08-01-corrected are permanently ambiguous.

Method, and its limits

Static inspection at fa66c64; probes that call the real _build_resilience_route_input and choose_resilience_route rather than re-implementing their logic; a read-only census of every route_audit/route_decision pair in the repository's ledgers. No tc run invoked, no ledger written, no synthetic execution created.

Stated plainly in the document rather than buried: measuring the timeout required five live local Ollama calls. They wrote nothing and touched no ledger, route, capability, privacy, or authority surface, but two findings depend on them.

Two deliberate non-assertions:

  • The 0/5 race result is scoped to one host, one cold model, one moment. The record explicitly does not claim the model path never wins anywhere — which is what makes the divergence a race rather than a constant.
  • The ~3015 ms cost of the nominal 1.5 s guard is measured but not explained. A refuted hypothesis (that resolve_model burns a second budget — it measures 0 ms, short-circuiting on an explicit model) is kept in the record, and the cause inside requests/urllib3 is left unisolated rather than guessed at.

Reconciliation against current main

main advanced three times while this was drafted. The record carries a reconciliation section rather than reading as current when part of it no longer is:

merge PR touched triage_core/? affects the record?
c798e0e #179 CR-DD-012B yes Yes — overlaps Finding 3
da96b66 #180 CR-DD-016 closeout no no
96eb3d3 #166 CR-132 no no

PR #179 landed CLASSIFICATION_REASON_CODES (governed_decision.py:65-72) distinguishing deterministic_classifier_default from deterministic_classifier_matchexactly the provenance Finding 3 reports as absent, so Finding 3 is recorded as partially superseded for the governed-decision path. It is not fully resolved: the ordinary tc run path still classifies without provenance (client.py:215) and still collapses bugfix/refactor (client.py:774-821). Whether the governed reason reaches persisted route evidence is left explicitly untraced, not asserted.

Findings 1, 2 and 4–7 stand: git diff c798e0e 96eb3d3 -- triage_core/ tests/ schemas/ is empty, and classifier.py, resilience_router.py, backends.py are byte-identical to fa66c64.

Test plan

  • Cumulative diff is exactly one new file under docs/operations/. No non-docs/ path touched; git status clean throughout.
  • All 25 file:line citations in the findings verified to resolve to the text attributed to them at fa66c64.
  • All reconciliation citations re-verified against 96eb3d3.
  • Citation drift re-checked using the two stale anchors as negative controlsrun_plan.py:59 and route_events.py:217 still resolve to the wrong content at 96eb3d3, confirming the drift is a real property of the record rather than an artifact of the checking method.
  • Router envelope derived by probing the real router across all 8 categories at the Aug-8 trial's recorded capability state, not by mirroring candidate logic.
  • Probe module provenance asserted at runtime (triage_core.__file__ pinned to this worktree) so results are not silently taken from the main checkout.
  • Verified the file is absent on main — clean add, no conflict.

No test-suite run: this PR adds a documentation record and changes no code.

What this does not grant

Merge authority for this PR. Any implementation change to the classifier or its maps. Any reopening of CR-DD-016, its Corrective Amendment, or its closeout. Any resolution of Track C, which remains untouched and separately scoped.

The record recommends two future CRs — fallback semantics/execution-path provenance, and classifier provenance/divergence — deliberately not combined, per the sequencing rule CR-DD-017 restates. Candidate 1 is explicitly narrowed so it does not re-scope what #179 already merged. Neither is proposed here.

🤖 Generated with Claude Code

coreytshaffer and others added 3 commits August 29, 2026 11:20
…igation

Read-only investigation record for Track B of CR-DD-016's Sequencing
section, which required that track stay read-only until its findings
warrant a separately scoped CR. This is that read-only output. No CR
identifier is minted, no implementation authority is requested or
granted, and no triage_core/ source, schema, test, or fixture changed.

Settles the three questions Track B was scoped to answer:

1. "refactor" IS the universal terminal fallback of both classify and
   classify_deterministic (classifier.py:58, :78).
2. It resolves PERMISSIVELY: sensitivity=low -> local_heavy with
   human_review_required=False. An unclassifiable prompt fails toward
   autonomous execution rather than toward review. This sharpens
   CR-DD-016's "semantically wrong" characterization -- the direction of
   the wrongness is the finding.
3. The Aug-8 trial motivating CR-DD-016 provably did NOT traverse the
   fallback, confirming that CR's Corrective Amendment by probe rather
   than by inference. The general question is unanswerable from recorded
   evidence by construction: bugfix and refactor both collapse onto
   task_class=code_repair (client.py:466, :468) before anything is
   written, so the two code_repair trials in 2026-08-01-corrected are
   permanently ambiguous.

Also records, as findings rather than conclusions: the model system
prompt and the regex cascade disagree on "format" (classifier.py:33 vs
:64), giving the same prompt opposite governance outcomes decided by a
race; the model path lost that race 5/5 on the measured host; and the
stated 1.5s guard costs ~3015ms wall clock.

Method and its limits stated in the document: static inspection at main
fa66c64, probes that call the real _build_resilience_route_input and
choose_resilience_route rather than mirroring them, read-only ledger
census, and five live local Ollama calls that wrote nothing. A refuted
hypothesis (that resolve_model burns a second timeout budget) is kept in
the record; it measures 0ms because it short-circuits on an explicit
model. The cause of the 2x timeout inside requests/urllib3 was NOT
isolated and is not asserted.

Recommends two separate future CRs -- fallback semantics/evidence, and
classifier provenance/divergence -- deliberately not combined, per the
sequencing rule CR-DD-017 restates. Track C remains untouched.

Documentation only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PR #179 (CR-DD-012B) merged while this record was being drafted, landing
work that overlaps Finding 3. Reconcile rather than let the record read
as current when part of it no longer is.

- New "Reconciliation against main at c798e0e" section: what #179 added
  (CLASSIFICATION_REASON_CODES with deterministic_classifier_match /
  _default at governed_decision.py:65-72, set via
  _classification_reason_code at run_plan.py:221-235/:468), and its
  effect on each finding.
- Finding 3 recorded as PARTIALLY SUPERSEDED for the governed-decision
  path only. The ordinary tc run path still classifies without
  provenance (client.py:215) and still collapses bugfix/refactor onto
  code_repair at sensitivity=low before route evidence is written
  (client.py:774-821). Whether the governed reason reaches persisted
  route evidence is left explicitly untraced, not asserted.
- Findings 1, 2 and 4-7 recorded as standing: classifier.py,
  resilience_router.py and backends.py are byte-identical between
  fa66c64 and c798e0e, and the client.py maps moved without changing
  content. Finding 7's --plan/execution seam arguably widens, since the
  plan path gained provenance execution still lacks.
- Citation drift stated plainly: references into client.py, run_plan.py
  and route_events.py were verified at fa66c64 and do NOT resolve at
  c798e0e; the other three files resolve at both.
- Deferred follow-up candidate 1 narrowed so a future CR does not
  re-scope work #179 already merged.

All nine new citations verified against origin/main. Documentation only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
main advanced twice more after the reconciliation was written against
c798e0e. Re-pin it to the current tip and record what each merge did.

- Reconciliation retitled to 96eb3d3, with a table of the three merges
  that landed while this record was drafted: #179 (CR-DD-012B, the only
  one to touch triage_core/), #180 (CR-DD-016 closeout, docs only), and
  #166 (CR-132, docs only -- one new CR document).
- Verified directly that `git diff c798e0e 96eb3d3 -- triage_core/
  tests/ schemas/` is empty, so no source fact changed across those two
  merges and every finding that held at c798e0e holds at 96eb3d3.
- All reconciliation citations re-verified against 96eb3d3.
- Citation drift re-checked using the two stale anchors as negative
  controls: run_plan.py:59 and route_events.py:217 still point at the
  wrong content at 96eb3d3, confirming the drift is a real property of
  the record rather than an artifact of how it was measured.

No finding changed. Documentation only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@netlify

netlify Bot commented Aug 29, 2026

Copy link
Copy Markdown

Deploy Preview for poetic-quokka-0fd859 ready!

Name Link
🔨 Latest commit eb1fadf
🔍 Latest deploy log https://app.netlify.com/projects/poetic-quokka-0fd859/deploys/6a933f61f96ff400085a6ebd
😎 Deploy Preview https://deploy-preview-181--poetic-quokka-0fd859.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coreytshaffer
coreytshaffer merged commit c0545a7 into main Aug 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant