feat(libsy): count classifier fail-open fallbacks on /metrics

[elyasmnvidian]

POST /v1/chat/completions
Content-Type: application/json

{"model":"switchyard/classifier","messages":[{"role":"user","content":"classify this"}]}

When the classifier's judge target returns HTTP 500, Switchyard still sends the request to the capable target. The request succeeds, but /metrics does not show that the classifier routed without a verdict:

$ curl -s localhost:4000/metrics | grep classifier_fail_open
# no output

Fix

A failed judge should not fail the caller's request. The classifier still routes without a verdict, and now increments switchyard_classifier_fail_open_total for each judge failure.

The counter has two labels: judge_model names the configured judge target, and reason is one of eight fixed categories. The code derives reason from the typed error and HTTP status only. It never uses request or response text as a label.

The non-5xx HTTP category is named upstream_non_5xx; an upstream error can carry any status below 500, not only a 4xx status.

Before and after

Before this change, the request succeeds but the metric query returns nothing. After the change, the same judge failure produces:

$ curl -s localhost:4000/metrics | grep classifier_fail_open
switchyard_classifier_fail_open_total{judge_model="judge-model",reason="upstream_5xx"} 1

A valid judge verdict does not increment the counter, even when it selects the same capable target as the fallback. Routing behavior and the caller's response are unchanged; this PR only adds the counter and its documentation. There is no performance claim.

How tested

cargo test -p switchyard-libsy client_errors_map_to_bounded_fail_open_reasons
cargo test -p switchyard-libsy --test observability classifier_fail_open_records_each_failure_stage

The first test covers all eight bounded reason categories, including call_error. The second drives the real LlmTaskClassifier through judge-call, stream-decoding, and verdict-parsing failures, then checks that a valid verdict is not counted.

[coderabbitai]

Walkthrough

Changes

The judge classifier now categorizes fail-open errors, logs them, and records a metric tagged by model and reason. Integration tests cover failure modes and confirm valid verdicts are not counted.

Classifier fail-open telemetry

Layer / File(s)	Summary
Failure categorization and reporting crates/libsy/src/algorithms/util/llm_judge.rs	Judge-call, aggregation, and parsing failures now use bounded reason labels while preserving fail-open behavior.
Fail-open metric recording crates/libsy/src/observability.rs	Adds the switchyard.classifier_fail_open counter with judge_model and reason attributes.
Failure-mode integration coverage crates/libsy/tests/observability.rs	Tests transport, HTTP, decoding, parsing, and valid-verdict scenarios with routing and metric assertions.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

A rabbit watched the judge reply,
Through broken roads and statuses high.
Each failure wore a reason tag,
While valid verdicts stayed unflagged.
Metrics hopped into the log—
“Fail open!” cheered the review-friend frog.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: counting classifier fail-open fallbacks on the metrics endpoint.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Fix failing CI checks

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

crates/libsy/tests/observability.rs (1)

858-863: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: no case covers timeout or the catch-all reasons.

client_error_reason maps Timeout → "timeout" and falls back to "client_error"/"call_error"; neither label is exercised. A JudgeOutcome::CallError(timeout_error) case would close the boundary most likely to be hit in production.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/libsy/tests/observability.rs` around lines 858 - 863, Extend the
observability test cases using the JudgeOutcome enum to cover
client_error_reason’s Timeout mapping and fallback labels. Add a
JudgeOutcome::CallError case that produces a timeout error and assert the
expected "timeout" reason, while also exercising the catch-all
client-error/call-error fallback as appropriate.

crates/libsy/src/observability.rs (1)

221-232: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Cache the fail-open counter handle
meter().u64_counter(...).build() runs on every fail-open, adding avoidable overhead on a hot outage path. A LazyLock<Counter<u64>> (or similar cached handle) would keep recording cheap; this test setup installs the global meter provider once before initialize_metrics(), so a process-wide handle should fit.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/libsy/src/observability.rs` around lines 221 - 232, Cache the counter
handle used by record_classifier_fail_open in a process-wide LazyLock (or
equivalent), initializing it with
meter().u64_counter("switchyard.classifier_fail_open").build() once. Update
record_classifier_fail_open to reuse the cached handle while preserving the
existing value and judge_model/reason labels.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/libsy/src/observability.rs`:
- Around line 221-232: Cache the counter handle used by
record_classifier_fail_open in a process-wide LazyLock (or equivalent),
initializing it with
meter().u64_counter("switchyard.classifier_fail_open").build() once. Update
record_classifier_fail_open to reuse the cached handle while preserving the
existing value and judge_model/reason labels.

In `@crates/libsy/tests/observability.rs`:
- Around line 858-863: Extend the observability test cases using the
JudgeOutcome enum to cover client_error_reason’s Timeout mapping and fallback
labels. Add a JudgeOutcome::CallError case that produces a timeout error and
assert the expected "timeout" reason, while also exercising the catch-all
client-error/call-error fallback as appropriate.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: daa96b57-cce4-48f5-a503-07972298ded6

📥 Commits

Reviewing files that changed from the base of the PR and between 7a4bb2a and 2becceb.

📒 Files selected for processing (3)

• crates/libsy/src/algorithms/util/llm_judge.rs
• crates/libsy/src/observability.rs
• crates/libsy/tests/observability.rs

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-205/
Built to branch gh-pages at 2026-08-04 17:50 UTC. Preview will be ready when the GitHub Pages deployment is complete.