[Bug] stage_router: capable_first can't offload at the default confidence_threshold=0.5

[joelagnel]

When benchmarking stage routing on TB-Lite with a frontier strong tier and a local small model weak tier, I reached for capable_first and left the threshold at its default.

The runs came back perfectly healthy: every request 200, no errors. But nothing ever went to the weak tier. Not a single turn.

After digging into the scorer I think this is a property of the shipped defaults rather than anything specific to our setup, so it seemed worth writing up in case others hit the same thing.

The short version

picker: capable_first (switchyard/lib/profiles/stage_router_config.py:76) and confidence_threshold: 0.5 (:81) are both defaults, and in combination the scorer can never select the efficient tier. You get there by omitting both knobs, which we suspect is the common path.

Where it comes from

In crates/libsy/src/algorithms/util/stage.rs (SIGNAL_UNIT=0.10, SCORE_GAIN=5.0, HARD_SEVERITY=0.7):

let raw = SIGNAL_UNIT
    * (d.severity / HARD_SEVERITY + d.spinning + d.exploring - d.production_intensity);
let score = (SCORE_GAIN * raw).tanh();
// confidence = score.abs();  gate is `if scored.confidence >= confidence_threshold`  (stage.rs:319)

Each maxed signal contributes one unit, so with k agreeing signals the confidence lands on a discrete ladder:

confidence = tanh(SCORE_GAIN × SIGNAL_UNIT × k) = tanh(0.5k)

k=1 → 0.4621     k=2 → 0.7616     k=3 → 0.9051

0.5 happens to sit in the gap between the k=1 and k=2 rungs.

That's fine in the escalation direction, because three terms push toward capable (severity, spinning, exploring) and any two of them reach k=2 = 0.7616. It's the de-escalation direction where it gets stuck: only one term is negative — production_intensity — and since it comes from ratio(...) it's bounded to [0, 1]. There's no second negative signal to stack with it, so the efficient side can never climb past the first rung:

max |confidence_negative| = |tanh(5.0 × 0.10 × (−1))| = tanh(0.5) = 0.4621171573

And capable_first only leaves its default tier when the score is confidently negative. So the 3-vs-1 asymmetry between the two directions is really what's going on here — which is also why efficient_first@0.5 behaves exactly as documented and only capable_first@0.5 gets stuck.

There's a near-miss that makes it easy to miss: before the squash, one maxed signal is exactly 5.0 × 0.10 = 0.5, which would satisfy the inclusive >= gate. The tanh compression pulls it down to 0.4621 — about 7.6% short.

What we measured

Two seeds of TB-Lite (20 tasks each, Hermes agent) at the default capable_first@0.5:

seed	tasks	weak-tier turns	weak tokens
1	20	0 / 217	0.0%
2	20	0 / 222	0.0%

439 consecutive turns, none routed to the efficient tier, with zero upstream errors in either run.

Changing only the threshold to 0.45 (same profile, same tasks) -- offload becomes 13.9% (token-weighted), and efficient_first@0.45 reaches 58.0%. So the tier plumbing is working fine; it's specifically the capable_first + >0.4621 combination that can't fire.

Worth noting the hard de-escalate shortcut (tests_passed && recent write/edit >= 1) didn't rescue it either — that path bypasses the scorer entirely, but across those 439 turns it never fired, so what you observe is a complete no-op rather than just a dead scorer branch.

The tricky part for users

The cliff sits at tanh(0.5) = 0.46211715726:

threshold	0.40	0.45	0.46	0.4621	0.4622	0.47	0.48	0.5
scorer offload fires?	yes	yes	yes	yes	no	no	no	no

Someone wondering "why isn't anything offloading?" would very reasonably nudge 0.5 → 0.48 → 0.47, still get nothing, and conclude the feature doesn't work. Nothing in the logs or in routing_decisions.stage_router separates "the signals were weak this turn" from "this branch cannot fire at all" -- every turn just shows up as no_signal / fall_open.

The docs get very close to this

docs/routing_algorithms/stage_router_routing.md:109 already spells out the key fact — one full signal scores ~0.46, "just under 0.5" — and draws exactly the right conclusion for escalation: a decisive escalation needs a strong signal plus corroboration. That reasoning is sound in the positive direction.

The subtlety is that the same sentence doesn't carry over to de-escalation, because there's only the one negative signal available to corroborate with. A few nearby places read a little differently once that's taken into account:

• :86 — "only drops to the efficient tier when the signals say 'efficient' and clear the threshold" — accurate in general, but at the recommended 0.5 that path can't be taken.
• :90 — "Raise it and only strong, decisive signals drop a turn to efficient" — reads as a continuous dial; above 0.4621 the rate isn't lower so much as fixed at zero.
• :101 — "Set 0.5 explicitly. It's the recommended starting point."
• :175 — "a 0.5 threshold takes ~1.5 signals of agreement — a policy that escalates ~20% of tasks maps roughly to confidence_threshold: 0.5 with capable_first" — with capable_first the scorer-driven action is de-escalation, so we weren't sure how to square this one with the above. Quite possibly it was calibrated with a classifier attached, in which case it may just need a note.

Possible fixes

Any one of these would have saved us the detour:

1. Lower the default confidence_threshold to 0.45.
2. Warn at config load when picker == capable_first && confidence_threshold > tanh(0.5).
3. Add a second negative signal so the efficient direction can reach the k=2 rung.

There's already a test pinning the constant — one_signal_scores_below_half (stage.rs:570) asserts 0.4621 < 0.5 — so the value is known and deliberate; it just doesn't currently flag the interaction with the default picker.

Happy to put up a PR for whichever direction you prefer, and happy to run any repro that would be useful. Thanks.