diff --git a/tools/spec-loop/.last-sync b/tools/spec-loop/.last-sync index b0bc41a0..9919ef91 100644 --- a/tools/spec-loop/.last-sync +++ b/tools/spec-loop/.last-sync @@ -1 +1 @@ -a1cff4441b93f8162aadb20a702b99437867d1db +de7d705468947fa7c1d37cd33405cd0d2741fed5 diff --git a/tools/spec-loop/specs/adapters.md b/tools/spec-loop/specs/adapters.md index cf61337a..37e6b0e4 100644 --- a/tools/spec-loop/specs/adapters.md +++ b/tools/spec-loop/specs/adapters.md @@ -90,7 +90,13 @@ by swapping the adapter, not the skill. - `tools/forwarder-relay/` — relay adapter for security reports forwarded by an upstream broker (e.g. the ASF security team); the counterpart to direct-intake adapters for the `security-issue-import-via-forwarder` - sub-skill. + sub-skill. Its `contact_handle` — who the skills address when proposing a + relay draft — defaults to a shared inbox declared org-level + (`organizations/ASF/organization.md` for the ASF profile) and inherited + through `project.md`, rather than naming an individual liaison. An adopter + whose relays do come through a named person overrides it per-project. The + multi-hop case, where a report reaches the project through more than one + broker, is designed in `docs/rfcs/RFC-AI-0008.md` and not yet implemented. - `tools/mail-patch/` and `tools/jira-patch/` — patch-over-mail / patch-over-Jira adapters; implement `contract:change-request` for projects that land patches via mailing-list review or Jira rather than diff --git a/tools/spec-loop/specs/adoption-and-setup.md b/tools/spec-loop/specs/adoption-and-setup.md index 443d1a6c..7f61a843 100644 --- a/tools/spec-loop/specs/adoption-and-setup.md +++ b/tools/spec-loop/specs/adoption-and-setup.md @@ -121,6 +121,16 @@ uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-valid ## Known gaps +- **Marketplace distribution is unspecified.** No spec covers the plugin + marketplace surface — the per-family manifests, the root catalogues, or the + `0.2.0.dev` version-stamping rule that `docs/setup/marketplaces.md` + documents. That rule is load-bearing: the marketplace is served from `main`, + so adopters do install dev versions, and `claude plugin update` compares + version strings rather than commit SHAs — a frozen suffix leaves an adopter + told they are "already at the latest version" indefinitely, recoverable only + by a full uninstall and reinstall. Worth its own spec rather than a bullet + here. + - `stable`; gaps appear as new agent targets to add to the registry ([`agents.md`](../../../skills/setup/agents.md)) or new override surfaces — recorded by the plan pass. diff --git a/tools/spec-loop/specs/issue-management-family.md b/tools/spec-loop/specs/issue-management-family.md index 91026b7e..e2761ce1 100644 --- a/tools/spec-loop/specs/issue-management-family.md +++ b/tools/spec-loop/specs/issue-management-family.md @@ -188,8 +188,16 @@ and a consistent propose-before-act discipline. ```bash uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-validate +PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \ + tools/skill-evals/evals/issue-triage/ \ + tools/skill-evals/evals/issue-reassess/ ``` +Every skill in the family ships a behavioural eval suite under +`tools/skill-evals/evals//`. Run `--cli` mode outside any sandbox that +denies the model CLI its credentials; an unauthenticated CLI now errors rather +than reporting a vacuous pass (see `meta-and-quality-tooling.md`). + ## Known gaps - **No adopter pilot has run the full family.** All eight skills are diff --git a/tools/spec-loop/specs/meta-and-quality-tooling.md b/tools/spec-loop/specs/meta-and-quality-tooling.md index 5afb296d..aabf48d4 100644 --- a/tools/spec-loop/specs/meta-and-quality-tooling.md +++ b/tools/spec-loop/specs/meta-and-quality-tooling.md @@ -37,7 +37,12 @@ trustworthy as it grows. `name`, `description`, `license`) and tool definitions, internal link integrity, placeholder conventions, license headers on tool Python files, and eval-coverage (soft check: warns when a skill has no eval suite). CLI: `skill-and-tool-validate`. -- `tools/skill-evals/` — harness for measuring skill behaviour. +- `tools/skill-evals/` — harness for measuring skill behaviour. A case whose + CLI produced no usable JSON reports ERROR unless something asserts on a + synthetic wrap key (`raw_output` / `stderr` / `exit_code`, via + `expected.json` or an `assertions.json` `field`) — a wrap nothing addresses + compares nothing, and passing it turned an unauthenticated CLI into a green + run. - `tools/sandbox-lint/` — lints the sandbox/permissions configuration. - `tools/symlink-lint/` — lints the framework's self-adoption skill symlinks: rejects cyclic symlinks, misdirected relays (canonical/ @@ -130,6 +135,9 @@ trustworthy as it grows. skills, and tools, and has its own tests. 8. `skill-evals` keeps mock tool output in the user turn and appends optional trusted repository context only to the system prompt. +9. `skill-evals` never reports PASS for a case in which nothing was graded: + a CLI that emits no JSON, or exits non-zero, errors unless the suite + explicitly asserts on the wrapped output. ## Validation diff --git a/tools/spec-loop/specs/project-agnosticism.md b/tools/spec-loop/specs/project-agnosticism.md index 6e902afd..3dcc7f3c 100644 --- a/tools/spec-loop/specs/project-agnosticism.md +++ b/tools/spec-loop/specs/project-agnosticism.md @@ -105,6 +105,15 @@ The three mechanisms, in order of preference: `dev-list-vote`) is incomplete; it must name at least one non-ASF option (`pr-approval`, `maintainer-roster`, `github-discussion`, …) and describe the adopter-facing default. +- **`` and `` are two placeholders, not one casing + choice.** `` is the display name (`Apache Foo`); `` is the + infrastructure slug used inside hostnames, addresses, and URL paths (`foo`). + Substituting one for the other yields a value that is wrong rather than + mis-cased. Because they differ only by case, a fixed-string lint written for + one does not catch the other, so `tools/dev/check-placeholders.sh` carries + both spellings and also matches spaced variants (`name: ""` as well + as `name:""`), since YAML, JSON, and GraphQL all accept either form + and pinning one lets the other through. - **Advisory, not paternalistic.** The audit surfaces candidate coupling for a maintainer to judge; some ASF strings are legitimate (examples, the ASF default profile, ASF-specific docs). It does not auto-rewrite. diff --git a/tools/spec-loop/specs/security-reporting.md b/tools/spec-loop/specs/security-reporting.md index 7ef9ca82..4347fc3d 100644 --- a/tools/spec-loop/specs/security-reporting.md +++ b/tools/spec-loop/specs/security-reporting.md @@ -50,6 +50,19 @@ health without navigating the tracker issue-by-issue. - **Config-driven.** Lifecycle category bands, time-to-triage signal, milestone vertical annotations, and the null-`upstream_repo` path are declared in the tool's `default-config.yaml` and overridden per-adopter. +- **The current bucket is projected to its end-of-bucket value.** A partially + elapsed month, quarter, or week is otherwise read as a genuine decline when + it is only incomplete. The projection is linear on the elapsed fraction and + splits by series kind: RATE series (counts accumulating from zero inside the + bucket — reports opened, reports rejected) project `observed / fraction` and + never below the observed count; LEVEL series (cumulative totals and + end-of-bucket snapshots, which carry over) extrapolate only the movement + within the bucket, `previous + (observed - previous) / fraction`, floored at + zero. Mean-based signals are deliberately not projected — a mean over the + items seen so far is already an estimate, not a partial accumulation. The + output states the elapsed fraction and the observed-to-projected pair, and + names the reason whenever projection is skipped (disabled, bucket already + complete, or no baseline bucket to project from). ## Out of scope