Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tools/spec-loop/.last-sync
Original file line number Diff line number Diff line change
@@ -1 +1 @@
a1cff4441b93f8162aadb20a702b99437867d1db
de7d705468947fa7c1d37cd33405cd0d2741fed5
8 changes: 7 additions & 1 deletion tools/spec-loop/specs/adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,13 @@ by swapping the adapter, not the skill.
- `tools/forwarder-relay/` — relay adapter for security reports forwarded
by an upstream broker (e.g. the ASF security team); the counterpart to
direct-intake adapters for the `security-issue-import-via-forwarder`
sub-skill.
sub-skill. Its `contact_handle` — who the skills address when proposing a
relay draft — defaults to a shared inbox declared org-level
(`organizations/ASF/organization.md` for the ASF profile) and inherited
through `project.md`, rather than naming an individual liaison. An adopter
whose relays do come through a named person overrides it per-project. The
multi-hop case, where a report reaches the project through more than one
broker, is designed in `docs/rfcs/RFC-AI-0008.md` and not yet implemented.
- `tools/mail-patch/` and `tools/jira-patch/` — patch-over-mail /
patch-over-Jira adapters; implement `contract:change-request` for
projects that land patches via mailing-list review or Jira rather than
Expand Down
10 changes: 10 additions & 0 deletions tools/spec-loop/specs/adoption-and-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,16 @@ uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-valid

## Known gaps

- **Marketplace distribution is unspecified.** No spec covers the plugin
marketplace surface — the per-family manifests, the root catalogues, or the
`0.2.0.dev<YYYYMMDDHHMM>` version-stamping rule that `docs/setup/marketplaces.md`
documents. That rule is load-bearing: the marketplace is served from `main`,
so adopters do install dev versions, and `claude plugin update` compares
version strings rather than commit SHAs — a frozen suffix leaves an adopter
told they are "already at the latest version" indefinitely, recoverable only
by a full uninstall and reinstall. Worth its own spec rather than a bullet
here.

- `stable`; gaps appear as new agent targets to add to the registry
([`agents.md`](../../../skills/setup/agents.md)) or new override
surfaces — recorded by the plan pass.
Expand Down
8 changes: 8 additions & 0 deletions tools/spec-loop/specs/issue-management-family.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,8 +188,16 @@ and a consistent propose-before-act discipline.

```bash
uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-validate
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
tools/skill-evals/evals/issue-triage/ \
tools/skill-evals/evals/issue-reassess/
```

Every skill in the family ships a behavioural eval suite under
`tools/skill-evals/evals/<skill>/`. Run `--cli` mode outside any sandbox that
denies the model CLI its credentials; an unauthenticated CLI now errors rather
than reporting a vacuous pass (see `meta-and-quality-tooling.md`).

## Known gaps

- **No adopter pilot has run the full family.** All eight skills are
Expand Down
10 changes: 9 additions & 1 deletion tools/spec-loop/specs/meta-and-quality-tooling.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,12 @@ trustworthy as it grows.
`name`, `description`, `license`) and tool definitions, internal link integrity,
placeholder conventions, license headers on tool Python files, and eval-coverage
(soft check: warns when a skill has no eval suite). CLI: `skill-and-tool-validate`.
- `tools/skill-evals/` — harness for measuring skill behaviour.
- `tools/skill-evals/` — harness for measuring skill behaviour. A case whose
CLI produced no usable JSON reports ERROR unless something asserts on a
synthetic wrap key (`raw_output` / `stderr` / `exit_code`, via
`expected.json` or an `assertions.json` `field`) — a wrap nothing addresses
compares nothing, and passing it turned an unauthenticated CLI into a green
run.
- `tools/sandbox-lint/` — lints the sandbox/permissions configuration.
- `tools/symlink-lint/` — lints the framework's self-adoption skill
symlinks: rejects cyclic symlinks, misdirected relays (canonical/
Expand Down Expand Up @@ -130,6 +135,9 @@ trustworthy as it grows.
skills, and tools, and has its own tests.
8. `skill-evals` keeps mock tool output in the user turn and appends optional
trusted repository context only to the system prompt.
9. `skill-evals` never reports PASS for a case in which nothing was graded:
a CLI that emits no JSON, or exits non-zero, errors unless the suite
explicitly asserts on the wrapped output.

## Validation

Expand Down
9 changes: 9 additions & 0 deletions tools/spec-loop/specs/project-agnosticism.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,15 @@ The three mechanisms, in order of preference:
`dev-list-vote`) is incomplete; it must name at least one non-ASF
option (`pr-approval`, `maintainer-roster`, `github-discussion`, …) and
describe the adopter-facing default.
- **`<PROJECT>` and `<project>` are two placeholders, not one casing
choice.** `<PROJECT>` is the display name (`Apache Foo`); `<project>` is the
infrastructure slug used inside hostnames, addresses, and URL paths (`foo`).
Substituting one for the other yields a value that is wrong rather than
mis-cased. Because they differ only by case, a fixed-string lint written for
one does not catch the other, so `tools/dev/check-placeholders.sh` carries
both spellings and also matches spaced variants (`name: "<project>"` as well
as `name:"<project>"`), since YAML, JSON, and GraphQL all accept either form
and pinning one lets the other through.
- **Advisory, not paternalistic.** The audit surfaces candidate coupling
for a maintainer to judge; some ASF strings are legitimate (examples,
the ASF default profile, ASF-specific docs). It does not auto-rewrite.
Expand Down
13 changes: 13 additions & 0 deletions tools/spec-loop/specs/security-reporting.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,19 @@ health without navigating the tracker issue-by-issue.
- **Config-driven.** Lifecycle category bands, time-to-triage signal,
milestone vertical annotations, and the null-`upstream_repo` path are
declared in the tool's `default-config.yaml` and overridden per-adopter.
- **The current bucket is projected to its end-of-bucket value.** A partially
elapsed month, quarter, or week is otherwise read as a genuine decline when
it is only incomplete. The projection is linear on the elapsed fraction and
splits by series kind: RATE series (counts accumulating from zero inside the
bucket — reports opened, reports rejected) project `observed / fraction` and
never below the observed count; LEVEL series (cumulative totals and
end-of-bucket snapshots, which carry over) extrapolate only the movement
within the bucket, `previous + (observed - previous) / fraction`, floored at
zero. Mean-based signals are deliberately not projected — a mean over the
items seen so far is already an estimate, not a partial accumulation. The
output states the elapsed fraction and the observed-to-projected pair, and
names the reason whenever projection is skipped (disabled, bucket already
complete, or no baseline bucket to project from).

## Out of scope

Expand Down