-
Notifications
You must be signed in to change notification settings - Fork 34
feat: add skill to author and maintain Compass manifests for agentic packs #154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
c625f31
89794ec
9a809a7
a3fe4b1
445a70e
fabfd5b
62a2f6c
00389a4
12d8efb
f5bc066
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,203 @@ | ||
| --- | ||
| name: compass-manifest-maintenance | ||
| description: | | ||
| Author and maintain Compass catalog-info.yaml manifests for agentic packs (skills, plugins, Locations, MCP inverse relations). Use when: | ||
| - Adding or updating a skill and its Compass manifest | ||
| - Registering an existing pack in Compass (manifests only; not creating a new agentic pack) | ||
| - MCP usage or orchestration changed in SKILL.md | ||
| - Auditing bidirectional dependsOn/dependencyOf drift against repo Compass conventions | ||
|
|
||
| File-based only: Read/Glob/Grep/Bash. For `.catalog/` marketplace metadata use create-collection. | ||
|
|
||
| NOT for: `.catalog/` metadata (use create-collection) or automated Compass registration. | ||
| model: inherit | ||
| color: magenta | ||
| license: Apache-2.0 | ||
| allowed-tools: Read Glob Grep Bash | ||
| --- | ||
|
|
||
| # /compass-manifest-maintenance Skill | ||
|
|
||
| **Audience:** Maintainers updating `catalog-info.yaml` manifests for Red Hat Compass (Backstage) registration. | ||
|
|
||
| **Goal:** Keep Compass manifests aligned with golden sources (`SKILL.md`, `mcps.json`, `AGENTS.md`) | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| **Required MCP Servers:** None — file-based skill. Do not use Compass MCP tools (`validate-entity`, `register-entity`, `get-catalog-entity`, `query-entity-graph`). | ||
|
|
||
| - Repository root as cwd. | ||
| - Read [CLAUDE.md](../../CLAUDE.md) Compass / Backstage Manifests section. | ||
| - Read [references/relationship-rules.md](references/relationship-rules.md) and [references/mcp-mapping.md](references/mcp-mapping.md). | ||
| - Templates: [assets/](assets/). | ||
|
|
||
| **Verification:** | ||
| ```bash | ||
| test -f CLAUDE.md && echo "✓ repo root" || echo "✗ wrong directory" | ||
| ``` | ||
|
|
||
| **Human Notification Protocol:** If drift audit finds violations, report file path and fix per workflow §1. | ||
|
|
||
| **Security:** Never display or expose credentials or token values. | ||
|
|
||
| ## When to Use This Skill | ||
|
|
||
| **Use when:** | ||
| - New skill needs `skills/<name>/catalog-info.yaml` and Location / inverse updates. | ||
| - Skill `allowed-tools` or orchestration changed → MCP or skill `dependsOn` must change. | ||
| - Register an existing pack in Compass (plugin, Location, root index, `system.yaml` manifests). | ||
| - Drift audit: skill on disk without manifest, stale `dependencyOf`, or dangling entity refs. | ||
| - After `agentic-contribution-skill` creates a skill (run before opening PR). | ||
|
|
||
| **Do not use when:** `.catalog/collection.yaml` (use **create-collection** instead), generic Compass platform tasks, or automated Compass registration. | ||
|
|
||
| ## Workflow | ||
|
|
||
| **MCP Tool:** None — file-based; uses Read, Glob, Grep, Bash only (see `allowed-tools` in frontmatter). | ||
|
|
||
| **Parameters:** N/A — no MCP tools; inputs are `<pack>`, `<skill-name>`, and on-disk manifest paths. | ||
|
|
||
| ### 1. Add or update a skill manifest | ||
|
|
||
| 1. **Resolve** `<pack>` and `<skill-name>` — confirm `<pack>/skills/<skill-name>/SKILL.md` exists. | ||
|
|
||
| 2. **Read golden sources** (precedence): | ||
| - `<pack>/<pack>-plugin.yaml` — `spec.lifecycle` (default for new skill manifest; see [relationship-rules.md](references/relationship-rules.md) Lifecycle) | ||
| - `SKILL.md` frontmatter: `name`, `description`, `allowed-tools` | ||
| - `SKILL.md` body: `Required MCP Servers`, `/skill-name` invocations, Dependencies, validator prerequisites | ||
| - `<pack>/mcps.json` — server keys (map via [mcp-mapping.md](references/mcp-mapping.md)) | ||
| - `<pack>/AGENTS.md` — tags/disciplines hints only (not orchestration inference) | ||
|
|
||
| 3. **Derive `dependsOn`** for the skill manifest: | ||
| - Always: `airesource:ai5-marketplace/<pack>` | ||
| - MCP: only servers the skill **actually uses** (tool prefixes + body); see [mcp-mapping.md](references/mcp-mapping.md) | ||
| - Other skills: orchestration or documented `/other-skill` invocations in `SKILL.md` | ||
| - Script-only skills: no `mcpserver:` entries | ||
|
|
||
| 4. **Set `spec.lifecycle`** (do not hardcode `beta`): | ||
| - Read `spec.lifecycle` from `<pack>/<pack>-plugin.yaml` — use as the **default** for the skill. | ||
| - **Human in the loop:** ask whether to change it. The skill may match the plugin or use a **less mature** value only (e.g. plugin `beta` → skill `development` is OK; plugin `development` → skill `beta` is **not** allowed). | ||
| - See [relationship-rules.md](references/relationship-rules.md) Lifecycle. | ||
|
|
||
| 5. **Write** `<pack>/skills/<skill-name>/catalog-info.yaml` from [assets/skill-catalog-info.yaml](assets/skill-catalog-info.yaml): | ||
| - `namespace: ai5-marketplace` | ||
| - `labels.distribution: external` | ||
| - `agents: []` | ||
| - `lifecycle:` value from step 4 (typically matches the plugin) | ||
| - `owner: group:redhat/ai5-marketplace` | ||
| - `backstage.io/source-location` → GitHub `main` branch SKILL.md URL | ||
| - `dependencyOf`: list orchestrators that `dependsOn` this skill (scan pack or update when editing orchestrator) | ||
|
|
||
| 6. **Update inverse manifests** ([relationship-rules.md](references/relationship-rules.md)): | ||
| - `<pack>/catalog-info.yaml` — `./skills/<skill-name>/catalog-info.yaml` in `spec.targets` | ||
| - `<pack>/<pack>-plugin.yaml` — `dependencyOf: airesource:ai5-marketplace/<skill-name>` | ||
| - Each `mcpserver:` in skill `dependsOn` → matching `mcps/*.yaml` `dependencyOf` | ||
| - Each skill in skill `dependsOn` → that skill's `dependencyOf` includes orchestrator | ||
|
|
||
| 7. **Reconcile plugin MCP deps** — plugin `dependsOn` = union of all `mcpserver:` refs across pack skill manifests. | ||
|
|
||
| ### 2. Register a new pack in Compass | ||
|
|
||
| 1. Create `<pack>/<pack>-plugin.yaml` from [assets/plugin-catalog-info.yaml](assets/plugin-catalog-info.yaml) with **`spec.lifecycle: development`** (default for new packs; confirm with user before raising maturity). | ||
| 2. Create `<pack>/catalog-info.yaml` from [assets/pack-location.yaml](assets/pack-location.yaml). | ||
| 3. Add `./<pack>/catalog-info.yaml` to root `catalog-info.yaml`. | ||
| 4. Add `airesource:ai5-marketplace/<pack>` to `system.yaml` `spec.dependencyOf`. | ||
| 5. Run **create-collection** for `<pack>/.catalog/` separately. | ||
|
|
||
| ### 3. Drift / compliance audit | ||
|
|
||
| Run `uv run python scripts/validate_compass_manifests.py` (or `make validate-compass-manifests`). It enforces the same structural rules as this table: | ||
|
|
||
| | Check | Rule | | ||
| |-------|------| | ||
| | Roster parity | Every `skills/*/SKILL.md` has `catalog-info.yaml` and pack Location target | | ||
| | Inverse parity | Plugin `dependencyOf` = full skill set; each skill `dependsOn` includes plugin | | ||
| | MCP inverse | Each skill `mcpserver:` in `dependsOn` → MCP `dependencyOf` includes skill | | ||
| | Skill inverse | Each skill→skill `dependsOn` → target `dependencyOf` includes source | | ||
| | Dangling refs | Every ref resolves to on-disk manifest or documented canonical MCP | | ||
| | Forbidden | No `partOf`/`hasPart` on AiResource/MCPServer; no redundant `dependsOn: system:default/agentic-plugins` on plugins | | ||
| | Namespace | Refs use `ai5-marketplace` except `mcpserver:redhat/*` and `default/agentic-plugins` | | ||
|
|
||
| Report violations with file path and fix per workflow §1. Do not weaken checks. | ||
|
|
||
| ### 4. Validate (file-based) | ||
|
|
||
| 1. Compare YAML against [assets/](assets/) and a known-good manifest in the same pack. | ||
| 2. Run `uv run python scripts/validate_compass_manifests.py` (included in `make validate` and `make validate-structure`). | ||
| 3. Tier 1 on this skill if edited: `uv run python scripts/validate_skills_tier1.py .claude/skills/compass-manifest-maintenance/SKILL.md`. | ||
| 4. Post-merge: maintainer may register in Compass UI manually (out of scope for this skill). | ||
|
|
||
| **Error Handling:** | ||
| - If skill on disk has no manifest → create from `assets/skill-catalog-info.yaml` and add Location target. | ||
| - If inverse `dependencyOf` missing on plugin or MCP → update per [relationship-rules.md](references/relationship-rules.md). | ||
| - If MCP ref cannot be resolved → grep pack manifests; do not invent new `mcpserver:` refs. | ||
|
|
||
| ## Self-review checklist | ||
|
|
||
| - [ ] Skill `spec.lifecycle` matches plugin default or a less mature value (never above the plugin). | ||
| - [ ] Manifest conventions: `agents: []`, `labels.distribution: external`, `namespace: ai5-marketplace`, `owner: group:redhat/ai5-marketplace`. | ||
| - [ ] MCP deps derived from `SKILL.md` usage, not copied from sibling skills. | ||
| - [ ] Every new `dependsOn` has matching `dependencyOf` on the target entity. | ||
| - [ ] Pack Location lists every skill manifest path. | ||
| - [ ] Plugin `dependencyOf` lists every skill in the pack. | ||
| - [ ] No `partOf`/`hasPart` on custom kinds. | ||
|
|
||
| ## Dependencies | ||
|
|
||
| ### Required MCP Servers | ||
|
|
||
| None — file-based manifest maintenance only. | ||
|
|
||
| ### Required MCP Tools | ||
|
|
||
| None — uses Read, Glob, Grep, Bash. | ||
|
|
||
| ### Related Skills | ||
|
|
||
| - **create-collection** — for `.catalog/` marketplace metadata (not Compass manifests) | ||
| - **agentic-contribution-skill** — run before this skill when creating a new skill | ||
|
|
||
| ### Reference Documentation | ||
|
|
||
| - [CLAUDE.md](../../CLAUDE.md) — entity kinds, namespaces, reference formats | ||
| - `scripts/validate_compass_manifests.py` — CI roster and bidirectional ref checks | ||
| - [references/relationship-rules.md](references/relationship-rules.md) | ||
| - [references/mcp-mapping.md](references/mcp-mapping.md) | ||
|
|
||
| ## Common Issues | ||
|
|
||
| - **COMPASS-1288 drift** — updated skill `dependsOn` but forgot plugin or MCP `dependencyOf`. | ||
| - **Blanket MCP deps** — declaring every MCP in `mcps.json` instead of per-skill `allowed-tools` usage. | ||
| - **Orchestration gaps** — `remediation`-style skills missing skill→skill edges or inverse `dependencyOf` on depended skills. | ||
| - **Canonical vs owned** — Lightspeed and Security use `mcpserver:redhat/...`; do not register duplicates in `mcps/`. | ||
| - **Unregistered packs** — `rh-developer`, `rh-ai-engineer`, `rh-automation` exist on disk but are not in root Location until explicitly added. | ||
|
|
||
| ## Example usage | ||
|
|
||
| ```bash | ||
| # CI structural validation (roster + bidirectional refs) | ||
| uv run python scripts/validate_compass_manifests.py | ||
| # or: make validate-compass-manifests | ||
|
|
||
| # Roster: skills on disk missing from Location (example: rh-sre) | ||
| comm -23 \ | ||
| <(find rh-sre/skills -name SKILL.md | sed 's|.*/skills/||;s|/SKILL.md||' | sort) \ | ||
| <(grep -oP 'skills/\K[^/]+(?=/catalog-info)' rh-sre/catalog-info.yaml | sort) | ||
|
|
||
| # Tier 1 lint for this maintenance skill | ||
| uv run python scripts/validate_skills_tier1.py .claude/skills/compass-manifest-maintenance/SKILL.md | ||
|
|
||
| # Full repo validation | ||
| make validate | ||
| ``` | ||
|
|
||
| Audit all four registered packs: | ||
|
|
||
| ```bash | ||
| for pack in ocp-admin rh-sre rh-virt rh-basic; do | ||
| echo "=== $pack ===" | ||
| comm -23 \ | ||
| <(find "$pack/skills" -name SKILL.md 2>/dev/null | sed 's|.*/skills/||;s|/SKILL.md||' | sort) \ | ||
| <(grep -oP 'skills/\K[^/]+(?=/catalog-info)' "$pack/catalog-info.yaml" 2>/dev/null | sort) | ||
| done | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Template: <pack>/catalog-info.yaml (Location) | ||
| apiVersion: backstage.io/v1alpha1 | ||
| kind: Location | ||
| metadata: | ||
| name: <pack> | ||
| title: <Pack Title> Agentic Pack | ||
| description: > | ||
| Points to all AI skill catalog entities in the <pack> agentic pack. | ||
| annotations: | ||
| agentic-plugins.redhat.com/version: "1.0" | ||
| agentic-plugins.redhat.com/repo: "agentic-plugins" | ||
| agentic.redhat.com/lola-module: "true" | ||
| agentic.redhat.com/lola-module-name: <pack> | ||
| agentic.redhat.com/repository: https://github.com/RHEcosystemAppEng/agentic-plugins | ||
| agentic.redhat.com/path: <pack> | ||
| spec: | ||
| targets: | ||
| - ./<pack>-plugin.yaml | ||
| - ./skills/<skill-name>/catalog-info.yaml | ||
| # List every skill manifest under skills/ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # Template: <pack>/<pack>-plugin.yaml | ||
| # Replace <pack>, title, description, tags, disciplines, categories, dependencyOf, dependsOn (MCP union). | ||
| # New packs: spec.lifecycle defaults to development (confirm with user before raising maturity). | ||
| apiVersion: backstage.io/v1alpha1 | ||
| kind: AiResource | ||
| metadata: | ||
| name: <pack> | ||
| namespace: ai5-marketplace | ||
| title: <Pack Title> | ||
| description: > | ||
| <Pack description from README or plugin manifest.> | ||
| tags: | ||
| - ai-skill | ||
| - <pack-tag> | ||
| annotations: | ||
| agentic-plugins.redhat.com/version: "1.0" | ||
| agentic-plugins.redhat.com/repo: "agentic-plugins" | ||
| agentic.redhat.com/lola-module: "true" | ||
| agentic.redhat.com/lola-module-name: <pack> | ||
| agentic.redhat.com/repository: https://github.com/RHEcosystemAppEng/agentic-plugins | ||
| agentic.redhat.com/path: <pack> | ||
| github.com/project-slug: RHEcosystemAppEng/agentic-plugins | ||
| links: | ||
| - url: https://github.com/RHEcosystemAppEng/agentic-plugins | ||
| title: Source Repository | ||
| icon: github | ||
| spec: | ||
| type: plugin | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. New skill in an existing pack: copy spec.lifecycle from the parent *-plugin.yaml. Do not hardcode lifecycle: beta in assets/skill-catalog-info.yaml, the compass-manifest-maintenance workflow, or CLAUDE.md. New pack: assets/plugin-catalog-info.yaml and the skill template must use the same default (development is the safer new-pack value).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @RedTanny I will take care of it in a separate PR for that ticket: https://redhat.atlassian.net/browse/APPENG-6307
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @rhartuv |
||
| lifecycle: development | ||
| owner: group:redhat/ai5-marketplace | ||
| system: default/agentic-plugins | ||
| disciplines: | ||
| - <discipline> | ||
| categories: | ||
| - <category> | ||
| agents: [] | ||
| dependsOn: | ||
| # Union of mcpserver refs used by any skill in this pack (omit if none) | ||
| # - mcpserver:ai5-marketplace/<owned-mcp> | ||
| # - mcpserver:redhat/<canonical-mcp> | ||
| dependencyOf: | ||
| - airesource:ai5-marketplace/<skill-name> | ||
| # Every skill in the pack — bidirectional with skill dependsOn plugin | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Template: skills/<skill-name>/catalog-info.yaml | ||
| # Replace <pack>, <skill-name>, title, description, tags, disciplines, categories, dependsOn. | ||
| # lifecycle: copy from <pack>/<pack>-plugin.yaml spec.lifecycle (default); may downgrade only — see relationship-rules.md | ||
| apiVersion: backstage.io/v1alpha1 | ||
| kind: AiResource | ||
| metadata: | ||
| name: <skill-name> | ||
| namespace: ai5-marketplace | ||
| title: <Skill Title> | ||
| description: > | ||
| <Multi-line description from SKILL.md frontmatter description, trimmed for catalog.> | ||
| labels: | ||
| distribution: external | ||
| annotations: | ||
| backstage.io/source-location: >- | ||
| url:https://github.com/RHEcosystemAppEng/agentic-plugins/blob/main/<pack>/skills/<skill-name>/SKILL.md | ||
| tags: | ||
| - ai-skill | ||
| - <domain-tag> | ||
| links: | ||
| - url: https://github.com/RHEcosystemAppEng/agentic-plugins | ||
| title: Source Repository | ||
| icon: github | ||
| spec: | ||
| type: skill | ||
| lifecycle: <plugin-lifecycle> # from <pack>/<pack>-plugin.yaml; same or less mature than plugin | ||
| owner: group:redhat/ai5-marketplace | ||
| disciplines: | ||
| - <discipline> | ||
| categories: | ||
| - <category> | ||
| agents: [] | ||
| dependsOn: | ||
| - airesource:ai5-marketplace/<pack> | ||
| # Add mcpserver: and airesource: (other skills) only when SKILL.md uses them | ||
| dependencyOf: [] | ||
| # dependencyOf: list orchestration skills that dependsOn this skill (COMPASS-1288) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # Deriving MCP `dependsOn` for skill manifests | ||
|
|
||
| Map **only** MCP servers the skill **actually uses**. Never copy a sibling manifest and never declare every key in `<pack>/mcps.json`. | ||
|
|
||
| ## Step 1 — Detect MCP usage in the skill | ||
|
|
||
| From `<pack>/skills/<skill>/SKILL.md`: | ||
|
|
||
| - `allowed-tools` in frontmatter (tool names / prefixes) | ||
| - `Required MCP Servers` / `Required MCP Tools` in the body | ||
| - Workflow steps that name an MCP server key or tool | ||
|
|
||
| If none of the above reference MCP tools → **no** `mcpserver:` in `dependsOn` (script-only, `WebFetch`, or public API skills). | ||
|
|
||
| ## Step 2 — Match usage to `mcps.json` keys | ||
|
|
||
| Read `<pack>/mcps.json`. Note which **server keys** the skill documents (e.g. `openshift-administration`, `lightspeed-mcp`, `aap-mcp-job-management`). | ||
|
|
||
| When `allowed-tools` lists tools but SKILL.md does not name a key, infer the server from context in the skill body (which MCP block documents those tools). | ||
|
|
||
| ## Step 3 — Resolve each key to a Compass entity ref | ||
|
|
||
| For each MCP the skill uses: | ||
|
|
||
| ### A. Owned MCP (manifest in this repo) | ||
|
|
||
| 1. List `mcps/*.yaml` (exclude `catalog-info.yaml`). | ||
| 2. Open manifests and read `metadata.name` and `metadata.namespace` (typically `ai5-marketplace`). | ||
| 3. Entity ref: `mcpserver:<namespace>/<metadata.name>`. | ||
| 4. **Deduplication:** Several `mcps.json` keys may share one Compass entity (same image/technology). Compare `mcps.json` entries (image URL, upstream repo in manifest `links`) to pick the correct owned manifest — do not create one ref per key if they map to the same `mcps/*.yaml`. | ||
| 5. Confirm the skill’s tools appear in that manifest’s `spec.primitives` (or match the server described in SKILL.md) when ambiguous. | ||
|
|
||
| ### B. Canonical MCP (registered upstream, not in `mcps/`) | ||
|
|
||
| If the pack references an MCP that **no** file under `mcps/` covers: | ||
|
|
||
| 1. Search existing skill manifests in the same pack (or repo) for the same `mcps.json` key or the same upstream product. | ||
| 2. Reuse the `mcpserver:` ref already used there (often `mcpserver:redhat/<name>`). | ||
| 3. Do **not** add a duplicate MCPServer manifest in `mcps/`. | ||
|
|
||
| When unsure, grep the repo for the `mcps.json` key or tool prefix in other `catalog-info.yaml` files under the same pack. | ||
|
|
||
| ## Step 4 — Update inverse relations | ||
|
|
||
| For each `mcpserver:` added to the skill’s `dependsOn`: | ||
|
|
||
| - Add `airesource:ai5-marketplace/<skill-name>` to that MCP manifest’s `spec.dependencyOf` (owned MCPs only — canonical MCPs are maintained by their owners). | ||
|
|
||
| ## Step 5 — Plugin MCP union | ||
|
|
||
| The pack plugin’s `dependsOn` MCP list = **union** of all `mcpserver:` refs across skill manifests in that pack — not the full set of keys in `mcps.json`. | ||
|
|
||
| ## Sanity checks | ||
|
|
||
| - Fewer MCP refs than keys in `mcps.json` is normal (per-skill usage). | ||
| - More MCP refs than the skill uses is wrong (blanket template deps). | ||
| - Canonical refs (`mcpserver:redhat/...`) do not get new files under `mcps/`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
isn't there any drift that we can prevent adding a CI validation script? (we already have many iof them in the scripts/ folder)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. I figured this was out of scope for this PR since it's just the skill and docs, but I can add a
scripts/validate_compass_manifests.pyscript for it. Let me know if you'd rather I include it here or open a separate ticket.