skills:eval cannot measure a skill reached from another skill

[cliffhall]

scripts/skill-eval.mjs runs every case with --max-turns 1, so it measures whether a skill is the model's first tool call. That is the right thing to measure for a skill a user reaches directly, and #2202 tuned every committed case to that shape. It leaves one class of invocation entirely unmeasured: a skill reached from inside another skill.

The gap, measured

testing ends its body with "See /test-servers for picking and building one", and test-servers is model-invocable, so the hand-off is live rather than a dead pointer. It still does not happen:

• Prompt: "Write an integration test that exercises tool listing against a real server."
• testing loaded first, 2/2.
• Across the next 9–12 tool calls in each run, test-servers was never loaded — the model went to Bash/Read/Glob instead.

The suite reports test-servers at 5/5, because all five of its cases ask for it directly. A skill that is only ever reachable through another would score a clean 100% while the hand-off silently never fires.

#2203 improves this particular hand-off two ways — test-servers' description now claims the "a change or a PR needs a real server to exercise it" situation, and testing's pointer moved from its last line into its lede — but neither is measured, and that is the point of this issue rather than a reason to close it.

There is also a task-prompt gap this would cover

"Can you smoke test this PR?" reaches no skill at all (0/3 before #2203's change, 0/3 after). That is arguably correct behavior: the model has to inspect the PR before it can know what fixture the change needs, so firing a skill on the first move would be firing blind. The realistic path for a prompt like that is a second-hop load once the investigation reveals a server is needed — exactly what nothing can currently observe.

What this would take

• A second eval mode with a higher turn budget that asserts an ordered pair — skill A fired, then skill B — rather than a single name.
• A decision about what its number means: a chain hit rate is not comparable to the first-move rate, so it probably wants reporting as its own column rather than folded into N/M cases at or above 80%.
• Read-only tool restriction has to hold across the extra turns; the current harness leans on --max-turns 1 for much of that containment.

Acceptance

• The suite can express and score "loading A should lead to loading B", with committed cases for at least the testing → test-servers hand-off.
• The chained measurement is reported distinctly from the first-move rate.
• docs/skill-authoring.md gains a section on when to write a chained case instead of a direct one.

[cliffhall]

Triage: Priority Low (total 5)

• Severity 2 — minor friction with a workaround: the hand-off can be strengthened and spot-checked by hand (as docs: tune skill eval cases and document how to write them #2203 does), the gap is that it cannot be measured
• Urgency 3 — wanted a milestone out; the cost of leaving it is that a chained-only skill can sit at a false 100% indefinitely
• Bonuses: none. Not a bug, no engagement yet, not assigned, not a sub-issue. Milestone bonus deliberately not claimed — filed through the create flow, so the milestone is the approval act rather than corroborating signal

Board: #28, Status Todo.

[cliffhall]

One refinement from probing while wiring up #2203, since it changes the design a little:

Some chains are same-turn, and those are already visible. Probing "How do I run the TUI against a server from a local build?" produced Skill(local-dev) and Skill(test-servers) in a single assistant turn on 2 of 3 samples — parallel tool calls, both captured by the existing collector.

So the gap is narrower than stated above: what is unmeasurable is specifically the sequential hop, where the model loads A, reads it, and only then decides it needs B. A chained mode may be able to score a useful fraction of hand-offs by asserting an unordered set of skills within the turn budget, which is a smaller change than ordered-pair tracking across turns.

[cliffhall]

Shipped in #2245 (merged as eb44e6d2).

All three acceptance criteria met:

• The suite can express and score "loading A should lead to loading B", with two committed cases for the testing → test-servers hand-off. A chain is scored as an ordered subsequence across strictly increasing assistant turns, so two Skill calls in one message — which cannot be causal, since the model has not seen the first skill's body — no longer count.
• The chained measurement is reported distinctly, in its own column with its own CHAIN_THRESHOLD (0.5, compared strictly) and CHAIN_MAX_TURNS (14). The two rates are never summed; a test asserts exactly two summary lines so nothing can silently merge them.
• docs/skill-authoring.md gained a section on when to write a chained case instead of a direct one — including the two ways such a case can false-pass, both found during review.

The measurement it immediately produced: the testing → test-servers pointer is followed 33–100% of the time at RUNS=3, which is the fact nothing could observe before. Strengthening it is #2247.