From 84af1355fc1b71b4a8d1c5082886c992db159f00 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Tue, 8 Sep 2026 03:51:48 +0200 Subject: [PATCH] chore(spec-loop): sync specs to main and advance .last-sync The marker sat at a1cff444, 17 commits behind main. Bumping it alone would claim those commits are described by the specs, so the drift is closed first. Specs updated for what actually shipped: - meta-and-quality-tooling: skill-evals errors, rather than passing, when a case's CLI produced no gradeable output (#1161). - security-reporting: the tracker dashboard projects the current partial bucket to its end-of-bucket value, splitting RATE series (accumulate from zero) from LEVEL series (carry over), and deliberately not projecting mean-based signals (#1158). - project-agnosticism: and are two placeholders holding different values, and the lint carries both spellings plus spaced variants (#1154). - adapters: the forwarder relay's contact_handle defaults to an org-level shared inbox rather than a named individual (#1135). The multi-hop coordinator case is designed in RFC-AI-0008 and unimplemented. - issue-management-family: the family's eval suites, and the note that --cli runs belong outside a credential-denying sandbox (#1145). Commits needing no spec change: #1152, #1143 and #1156 updated their own specs in-commit; #1149, #1147, #1151 are behaviour-preserving bug fixes; #1155 and #1141 are CI and dependency chores; #1159's spec edits landed with it; #1144 removes hardcoded literals that no spec asserted. One genuine gap recorded rather than papered over: no spec covers marketplace distribution or the dev-version stamping rule from #1160, which is load-bearing because `claude plugin update` compares version strings, not commit SHAs. Logged in adoption-and-setup as wanting its own spec. Generated-by: Claude Code (Opus 5) --- tools/spec-loop/.last-sync | 2 +- tools/spec-loop/specs/adapters.md | 8 +++++++- tools/spec-loop/specs/adoption-and-setup.md | 10 ++++++++++ tools/spec-loop/specs/issue-management-family.md | 8 ++++++++ tools/spec-loop/specs/meta-and-quality-tooling.md | 10 +++++++++- tools/spec-loop/specs/project-agnosticism.md | 9 +++++++++ tools/spec-loop/specs/security-reporting.md | 13 +++++++++++++ 7 files changed, 57 insertions(+), 3 deletions(-) diff --git a/tools/spec-loop/.last-sync b/tools/spec-loop/.last-sync index b0bc41a09..9919ef91f 100644 --- a/tools/spec-loop/.last-sync +++ b/tools/spec-loop/.last-sync @@ -1 +1 @@ -a1cff4441b93f8162aadb20a702b99437867d1db +de7d705468947fa7c1d37cd33405cd0d2741fed5 diff --git a/tools/spec-loop/specs/adapters.md b/tools/spec-loop/specs/adapters.md index cf61337a7..37e6b0e40 100644 --- a/tools/spec-loop/specs/adapters.md +++ b/tools/spec-loop/specs/adapters.md @@ -90,7 +90,13 @@ by swapping the adapter, not the skill. - `tools/forwarder-relay/` — relay adapter for security reports forwarded by an upstream broker (e.g. the ASF security team); the counterpart to direct-intake adapters for the `security-issue-import-via-forwarder` - sub-skill. + sub-skill. Its `contact_handle` — who the skills address when proposing a + relay draft — defaults to a shared inbox declared org-level + (`organizations/ASF/organization.md` for the ASF profile) and inherited + through `project.md`, rather than naming an individual liaison. An adopter + whose relays do come through a named person overrides it per-project. The + multi-hop case, where a report reaches the project through more than one + broker, is designed in `docs/rfcs/RFC-AI-0008.md` and not yet implemented. - `tools/mail-patch/` and `tools/jira-patch/` — patch-over-mail / patch-over-Jira adapters; implement `contract:change-request` for projects that land patches via mailing-list review or Jira rather than diff --git a/tools/spec-loop/specs/adoption-and-setup.md b/tools/spec-loop/specs/adoption-and-setup.md index 443d1a6c3..7f61a8438 100644 --- a/tools/spec-loop/specs/adoption-and-setup.md +++ b/tools/spec-loop/specs/adoption-and-setup.md @@ -121,6 +121,16 @@ uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-valid ## Known gaps +- **Marketplace distribution is unspecified.** No spec covers the plugin + marketplace surface — the per-family manifests, the root catalogues, or the + `0.2.0.dev` version-stamping rule that `docs/setup/marketplaces.md` + documents. That rule is load-bearing: the marketplace is served from `main`, + so adopters do install dev versions, and `claude plugin update` compares + version strings rather than commit SHAs — a frozen suffix leaves an adopter + told they are "already at the latest version" indefinitely, recoverable only + by a full uninstall and reinstall. Worth its own spec rather than a bullet + here. + - `stable`; gaps appear as new agent targets to add to the registry ([`agents.md`](../../../skills/setup/agents.md)) or new override surfaces — recorded by the plan pass. diff --git a/tools/spec-loop/specs/issue-management-family.md b/tools/spec-loop/specs/issue-management-family.md index 91026b7e6..e2761ce1d 100644 --- a/tools/spec-loop/specs/issue-management-family.md +++ b/tools/spec-loop/specs/issue-management-family.md @@ -188,8 +188,16 @@ and a consistent propose-before-act discipline. ```bash uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-validate +PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \ + tools/skill-evals/evals/issue-triage/ \ + tools/skill-evals/evals/issue-reassess/ ``` +Every skill in the family ships a behavioural eval suite under +`tools/skill-evals/evals//`. Run `--cli` mode outside any sandbox that +denies the model CLI its credentials; an unauthenticated CLI now errors rather +than reporting a vacuous pass (see `meta-and-quality-tooling.md`). + ## Known gaps - **No adopter pilot has run the full family.** All eight skills are diff --git a/tools/spec-loop/specs/meta-and-quality-tooling.md b/tools/spec-loop/specs/meta-and-quality-tooling.md index 5afb296d6..aabf48d48 100644 --- a/tools/spec-loop/specs/meta-and-quality-tooling.md +++ b/tools/spec-loop/specs/meta-and-quality-tooling.md @@ -37,7 +37,12 @@ trustworthy as it grows. `name`, `description`, `license`) and tool definitions, internal link integrity, placeholder conventions, license headers on tool Python files, and eval-coverage (soft check: warns when a skill has no eval suite). CLI: `skill-and-tool-validate`. -- `tools/skill-evals/` — harness for measuring skill behaviour. +- `tools/skill-evals/` — harness for measuring skill behaviour. A case whose + CLI produced no usable JSON reports ERROR unless something asserts on a + synthetic wrap key (`raw_output` / `stderr` / `exit_code`, via + `expected.json` or an `assertions.json` `field`) — a wrap nothing addresses + compares nothing, and passing it turned an unauthenticated CLI into a green + run. - `tools/sandbox-lint/` — lints the sandbox/permissions configuration. - `tools/symlink-lint/` — lints the framework's self-adoption skill symlinks: rejects cyclic symlinks, misdirected relays (canonical/ @@ -130,6 +135,9 @@ trustworthy as it grows. skills, and tools, and has its own tests. 8. `skill-evals` keeps mock tool output in the user turn and appends optional trusted repository context only to the system prompt. +9. `skill-evals` never reports PASS for a case in which nothing was graded: + a CLI that emits no JSON, or exits non-zero, errors unless the suite + explicitly asserts on the wrapped output. ## Validation diff --git a/tools/spec-loop/specs/project-agnosticism.md b/tools/spec-loop/specs/project-agnosticism.md index 6e902afd0..3dcc7f3cd 100644 --- a/tools/spec-loop/specs/project-agnosticism.md +++ b/tools/spec-loop/specs/project-agnosticism.md @@ -105,6 +105,15 @@ The three mechanisms, in order of preference: `dev-list-vote`) is incomplete; it must name at least one non-ASF option (`pr-approval`, `maintainer-roster`, `github-discussion`, …) and describe the adopter-facing default. +- **`` and `` are two placeholders, not one casing + choice.** `` is the display name (`Apache Foo`); `` is the + infrastructure slug used inside hostnames, addresses, and URL paths (`foo`). + Substituting one for the other yields a value that is wrong rather than + mis-cased. Because they differ only by case, a fixed-string lint written for + one does not catch the other, so `tools/dev/check-placeholders.sh` carries + both spellings and also matches spaced variants (`name: ""` as well + as `name:""`), since YAML, JSON, and GraphQL all accept either form + and pinning one lets the other through. - **Advisory, not paternalistic.** The audit surfaces candidate coupling for a maintainer to judge; some ASF strings are legitimate (examples, the ASF default profile, ASF-specific docs). It does not auto-rewrite. diff --git a/tools/spec-loop/specs/security-reporting.md b/tools/spec-loop/specs/security-reporting.md index 7ef9ca828..4347fc3db 100644 --- a/tools/spec-loop/specs/security-reporting.md +++ b/tools/spec-loop/specs/security-reporting.md @@ -50,6 +50,19 @@ health without navigating the tracker issue-by-issue. - **Config-driven.** Lifecycle category bands, time-to-triage signal, milestone vertical annotations, and the null-`upstream_repo` path are declared in the tool's `default-config.yaml` and overridden per-adopter. +- **The current bucket is projected to its end-of-bucket value.** A partially + elapsed month, quarter, or week is otherwise read as a genuine decline when + it is only incomplete. The projection is linear on the elapsed fraction and + splits by series kind: RATE series (counts accumulating from zero inside the + bucket — reports opened, reports rejected) project `observed / fraction` and + never below the observed count; LEVEL series (cumulative totals and + end-of-bucket snapshots, which carry over) extrapolate only the movement + within the bucket, `previous + (observed - previous) / fraction`, floored at + zero. Mean-based signals are deliberately not projected — a mean over the + items seen so far is already an estimate, not a partial accumulation. The + output states the elapsed fraction and the observed-to-projected pair, and + names the reason whenever projection is skipped (disabled, bucket already + complete, or no baseline bucket to project from). ## Out of scope