diff --git a/experiments/README.md b/experiments/README.md new file mode 100644 index 0000000..13eb1f4 --- /dev/null +++ b/experiments/README.md @@ -0,0 +1,17 @@ +# Experiments + +Scenario harnesses for exercising SEP-2640 v1 (`skills/list` + `skills/get` + `resources/directory/read`) against real hosts and servers. This tree was restarted from scratch for the v1 protocol methods; the earlier multi-host harnesses (fast-agent, hermes, HF demos) built against the index.json draft live on the `feature/resource-sep-early-findings` branch for reference. + +Current harnesses: + +- [`goose/`](goose/) — drives the goose CLI (branch `skills-sep-2640-port` of https://github.com/olaservo/goose) against a running SEP-2640 server. + +Reference server for all harnesses: [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `feature/agent-skills-v2` of https://github.com/olaservo/github-mcp-server. + +![Skill surfaces of github-mcp-server on feature/agent-skills-v2](sep-2640-github-mcp-server-surfaces.svg) + +The 28 bundled skills are plain `SKILL.md` directories in the checkout, embedded at build time and served as `skill://github//…`; `skills/list` enumerates only these. Skills in any GitHub repository are reachable by URI (`skill://{owner}/{repo}/{skill}/{file}`) through `skills/get`, `resources/read`, and `resources/directory/read`, with `list_repo_skills` as the tool that finds them, but they never appear in `skills/list`. + +Second server: [skills-over-mcp-demo](https://github.com/olaservo/skills-over-mcp-demo), a small SEP-2640 server on the v2 TypeScript SDK, live and unauthenticated at `https://olaservo-skills-over-mcp-demo.hf.space/mcp`. It adds what the GitHub server lacks: an unlisted skill reachable only through the server's `instructions` pointer and `skills/get`, supporting files (so the digest and size gate is exercised), and a multi-segment skill path. Free Spaces sleep when idle; the harness's wire probe wakes it before goose connects. + +Conventions shared by all harnesses: the runner never spawns the MCP server — it connects to an already-running server at the scenario's `mcp_server.endpoint` — and the scenario file is always an explicit `--scenario ` argument. diff --git a/experiments/goose/README.md b/experiments/goose/README.md new file mode 100644 index 0000000..5a4a418 --- /dev/null +++ b/experiments/goose/README.md @@ -0,0 +1,48 @@ +# goose scenario harness + +Runs one scenario against the goose CLI and a running SEP-2640 server, and records two things: what was **discoverable** (the server's declared capability and `skills/list` names, probed directly on the wire, plus whether the model could enumerate the skills from its own instructions) and what was **used** (every tool call goose made, which skills were loaded, which resources were read). + +## Surfaces under test + +![Skills, MCP-served skills, MCP resources, and MCP tools in goose](sep-2640-goose-surfaces.svg) + +How goose (`skills-sep-2640-port`) exposes SEP-2640 to the model: local and MCP-served skills share one prompt list and one `load_skill` tool that routes by origin; MCP loads pass through the verification gate (sha256 + size per file, frontmatter identity, unlisted-file refusal, one `skills/get` retry); `read_resource` reads raw content from any server with no verification or skill activation; prefixed MCP tools pass straight through to `tools/call`. `"dynamic"` entries skip the gate. + +## Prerequisites + +- A goose build with skills-over-MCP support: branch `skills-sep-2640-port` of https://github.com/olaservo/goose (`cargo build -p goose-cli`). Point the harness at the binary with `--goose` or `GOOSE_BIN`. Use `fdf9ae6c` or later: the 2026-08-21/24 upstream merges left `OAuthStepUpClient` without `skills_list`/`skills_get`/`directory_read` forwarding, so every streamable HTTP server reported `Transport closed`. +- A running SEP-2640 server. Reference: draft PR [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `olaservo:feature/agent-skills-v2` (`09ea2f9f` or later for the b405ba5 `size`/`"dynamic"` contract) — `go build -o github-mcp-server-skills.exe ./cmd/github-mcp-server`, then `./github-mcp-server-skills.exe http --port 8082`. The harness never starts the server; it connects to the scenario's `mcp_server.endpoint`. +- Auth: when the scenario sets `mcp_server.bearer_cmd`, it is run once and its output sent as a `Authorization: Bearer …` header by both the wire probe and goose. Omit it for unauthenticated servers. +- An LLM provider goose can use. The harness writes an isolated goose config (nothing in your real config is touched); provider credentials come from the system keyring or provider env vars as usual. Defaults: `anthropic` / `claude-sonnet-4-6`, overridable per scenario. + +Scenarios named `demo-*` run against the public demo Space instead (see [`../README.md`](../README.md)); they need no server or auth setup. Observed 2026-08-25: the live Space negotiates `2025-11-25` and its `skills/list` carries no `ttlMs`/`cacheScope`, unlike the demo repo's README claims for its current commit; the deployment may be older than the repo. + +## Run + +``` +uv run --with pyyaml run_scenario.py --scenario scenarios/discovery-and-load.yaml +``` + +The result JSON lands in `results/`, the isolated goose config root in a temp dir (path printed; it contains the bearer token when one was used, delete when done). Exit code is 0 only if every check passed. + +## Scenario format + +```yaml +name: discovery-and-load +mcp_server: + endpoint: http://localhost:8082/mcp + bearer_cmd: gh auth token # optional; omit for unauthenticated servers +provider: anthropic # optional +model: claude-sonnet-4-6 # optional +prompt: | + ...single user turn given to goose... +expect: + extension_declared: true # server declares io.modelcontextprotocol/skills + resources_sized: true # every manifest element in skills/list carries an integer size + discovered: [create-issue] # ⊆ wire skills/list names AND each appears in the reply text + loaded: [create-issue] # ⊆ load_skill calls whose result was framed content (`# Loaded Skill:` / `# Loaded:`), not an error + tools: [load_skill] # optional, ⊆ tool names goose called + resources_read: [skill://…] # optional, ⊆ the `uri` args of read_resource calls +``` + +Every `expect` key is optional; omitted keys are recorded but not asserted. The harness grades nothing else — it logs the full tool-call trace and the reply tail so a human can read what actually happened. diff --git a/experiments/goose/results/demo-discovery-and-load.20260825-205208.json b/experiments/goose/results/demo-discovery-and-load.20260825-205208.json new file mode 100644 index 0000000..5d83915 --- /dev/null +++ b/experiments/goose/results/demo-discovery-and-load.20260825-205208.json @@ -0,0 +1,132 @@ +{ + "scenario": "demo-discovery-and-load", + "timestamp": "2026-08-25T20:52:08", + "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "tabletop-dice", + "uri": "skill://dice-roller/tabletop-dice/SKILL.md", + "resources": 2, + "sized": true + }, + { + "name": "mcp-glossary", + "uri": "skill://mcp-glossary/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "release-notes-writer", + "uri": "skill://release-notes-writer/SKILL.md", + "resources": 2, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "tabletop-dice" + } + } + ], + "load_skill_calls": [ + "tabletop-dice" + ], + "skills_loaded": [ + "tabletop-dice" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-pkf8edyi\\wd\n L L goose is ready\nHere are the skills listed as coming from connected MCP servers in my system instructions:\n\n- mcp-glossary (scenario_server)\n- release-notes-writer (scenario_server)\n- tabletop-dice (scenario_server)\n\n---\n\nNow loading the skill:\n ────────────────────────────────────────\n ▸ load_skill\n name: tabletop-dice\n\nThe first markdown heading line of the loaded content is:\n\n**`# Tabletop Dice`**\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "resources_sized", + "pass": true, + "detail": { + "manifest_entries": 3, + "unsized": [] + } + }, + { + "check": "discovered on wire: tabletop-dice", + "pass": true, + "detail": [ + "tabletop-dice", + "mcp-glossary", + "release-notes-writer" + ] + }, + { + "check": "enumerated by model: tabletop-dice", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: mcp-glossary", + "pass": true, + "detail": [ + "tabletop-dice", + "mcp-glossary", + "release-notes-writer" + ] + }, + { + "check": "enumerated by model: mcp-glossary", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: release-notes-writer", + "pass": true, + "detail": [ + "tabletop-dice", + "mcp-glossary", + "release-notes-writer" + ] + }, + { + "check": "enumerated by model: release-notes-writer", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "loaded: tabletop-dice", + "pass": true, + "detail": { + "loaded": [ + "tabletop-dice" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/results/demo-supporting-file.20260825-205332.json b/experiments/goose/results/demo-supporting-file.20260825-205332.json new file mode 100644 index 0000000..e93e8ab --- /dev/null +++ b/experiments/goose/results/demo-supporting-file.20260825-205332.json @@ -0,0 +1,102 @@ +{ + "scenario": "demo-supporting-file", + "timestamp": "2026-08-25T20:53:32", + "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "tabletop-dice", + "uri": "skill://dice-roller/tabletop-dice/SKILL.md", + "resources": 2, + "sized": true + }, + { + "name": "mcp-glossary", + "uri": "skill://mcp-glossary/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "release-notes-writer", + "uri": "skill://release-notes-writer/SKILL.md", + "resources": 2, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "tabletop-dice" + } + }, + { + "tool": "load_skill", + "args": { + "name": "tabletop-dice/references/dice-notation.md" + } + } + ], + "load_skill_calls": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "skills_loaded": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-eoa6k3on\\wd\n L L goose is ready\nI'll load the tabletop-dice skill first, then follow its instructions for loading the supporting file.\n ────────────────────────────────────────\n ▸ load_skill\n name: tabletop-dice\n\nNow I'll load the supporting file as instructed:\n ────────────────────────────────────────\n ▸ load_skill\n name: tabletop-dice/references/dice-notation.md\n\nThe first markdown heading line of the supporting file is:\n\n```\n# Dice Notation Reference\n```\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "loaded: tabletop-dice", + "pass": true, + "detail": { + "loaded": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "failed": [] + } + }, + { + "check": "loaded: tabletop-dice/references/dice-notation.md", + "pass": true, + "detail": { + "loaded": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/results/demo-unlisted-skill.20260825-205215.json b/experiments/goose/results/demo-unlisted-skill.20260825-205215.json new file mode 100644 index 0000000..14620f3 --- /dev/null +++ b/experiments/goose/results/demo-unlisted-skill.20260825-205215.json @@ -0,0 +1,82 @@ +{ + "scenario": "demo-unlisted-skill", + "timestamp": "2026-08-25T20:52:15", + "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "tabletop-dice", + "uri": "skill://dice-roller/tabletop-dice/SKILL.md", + "resources": 2, + "sized": true + }, + { + "name": "mcp-glossary", + "uri": "skill://mcp-glossary/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "release-notes-writer", + "uri": "skill://release-notes-writer/SKILL.md", + "resources": 2, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "skill://secret-menu/SKILL.md" + } + } + ], + "load_skill_calls": [ + "skill://secret-menu/SKILL.md" + ], + "skills_loaded": [ + "skill://secret-menu/SKILL.md" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-4l0b61r8\\wd\n L L goose is ready\n\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://secret-menu/SKILL.md\n\nThe first markdown heading line of the loaded content is:\n\n```\n# Secret Menu\n```\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "loaded: skill://secret-menu/SKILL.md", + "pass": true, + "detail": { + "loaded": [ + "skill://secret-menu/SKILL.md" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/results/discovery-and-load.20260827-190206.json b/experiments/goose/results/discovery-and-load.20260827-190206.json new file mode 100644 index 0000000..2bfb6a9 --- /dev/null +++ b/experiments/goose/results/discovery-and-load.20260827-190206.json @@ -0,0 +1,357 @@ +{ + "scenario": "discovery-and-load", + "timestamp": "2026-08-27T19:02:06", + "endpoint": "http://localhost:8082/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "address-pr-feedback", + "uri": "skill://github/address-pr-feedback/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "browse-discussions", + "uri": "skill://github/browse-discussions/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "contribute-oss", + "uri": "skill://github/contribute-oss/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "create-issue", + "uri": "skill://github/create-issue/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "create-pr", + "uri": "skill://github/create-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "debug-ci", + "uri": "skill://github/debug-ci/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "delegate-to-copilot", + "uri": "skill://github/delegate-to-copilot/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "discover-github", + "uri": "skill://github/discover-github/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "discover-mcp-skills", + "uri": "skill://github/discover-mcp-skills/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "explore-repo", + "uri": "skill://github/explore-repo/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "fix-dependabot", + "uri": "skill://github/fix-dependabot/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "get-context", + "uri": "skill://github/get-context/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "handle-notifications", + "uri": "skill://github/handle-notifications/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-labels", + "uri": "skill://github/manage-labels/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-project", + "uri": "skill://github/manage-project/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-repo", + "uri": "skill://github/manage-repo/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-sub-issues", + "uri": "skill://github/manage-sub-issues/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "merge-pr", + "uri": "skill://github/merge-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "prepare-release", + "uri": "skill://github/prepare-release/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "research-vulnerability", + "uri": "skill://github/research-vulnerability/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "review-pr", + "uri": "skill://github/review-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "search-code", + "uri": "skill://github/search-code/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "security-audit", + "uri": "skill://github/security-audit/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "self-review-pr", + "uri": "skill://github/self-review-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "share-snippet", + "uri": "skill://github/share-snippet/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "trace-history", + "uri": "skill://github/trace-history/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "triage-issues", + "uri": "skill://github/triage-issues/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "trigger-workflow", + "uri": "skill://github/trigger-workflow/SKILL.md", + "resources": 1, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "create-issue" + } + } + ], + "load_skill_calls": [ + "create-issue" + ], + "skills_loaded": [ + "create-issue" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260828_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-quott2po\\wd\n L L goose is ready\nHere are the skills from connected MCP servers, exactly as written in my system instructions:\n\n- address-pr-feedback (scenario_server)\n- browse-discussions (scenario_server)\n- contribute-oss (scenario_server)\n- create-issue (scenario_server)\n- create-pr (scenario_server)\n- debug-ci (scenario_server)\n- delegate-to-copilot (scenario_server)\n- discover-github (scenario_server)\n- discover-mcp-skills (scenario_server)\n- explore-repo (scenario_server)\n- fix-dependabot (scenario_server)\n- get-context (scenario_server)\n- handle-notifications (scenario_server)\n- manage-labels (scenario_server)\n- manage-project (scenario_server)\n- manage-repo (scenario_server)\n- manage-sub-issues (scenario_server)\n- merge-pr (scenario_server)\n- prepare-release (scenario_server)\n- research-vulnerability (scenario_server)\n- review-pr (scenario_server)\n- search-code (scenario_server)\n- security-audit (scenario_server)\n- self-review-pr (scenario_server)\n- share-snippet (scenario_server)\n- trace-history (scenario_server)\n- triage-issues (scenario_server)\n- trigger-workflow (scenario_server)\n\n---\n\nNow loading the `create-issue` skill:\n ────────────────────────────────────────\n ▸ load_skill\n name: create-issue\n\nThe first markdown heading line of the loaded content is:\n\n**`# Create Issue`**\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "resources_sized", + "pass": true, + "detail": { + "manifest_entries": 28, + "unsized": [] + } + }, + { + "check": "discovered on wire: create-issue", + "pass": true, + "detail": [ + "address-pr-feedback", + "browse-discussions", + "contribute-oss", + "create-issue", + "create-pr", + "debug-ci", + "delegate-to-copilot", + "discover-github", + "discover-mcp-skills", + "explore-repo", + "fix-dependabot", + "get-context", + "handle-notifications", + "manage-labels", + "manage-project", + "manage-repo", + "manage-sub-issues", + "merge-pr", + "prepare-release", + "research-vulnerability", + "review-pr", + "search-code", + "security-audit", + "self-review-pr", + "share-snippet", + "trace-history", + "triage-issues", + "trigger-workflow" + ] + }, + { + "check": "enumerated by model: create-issue", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: create-pr", + "pass": true, + "detail": [ + "address-pr-feedback", + "browse-discussions", + "contribute-oss", + "create-issue", + "create-pr", + "debug-ci", + "delegate-to-copilot", + "discover-github", + "discover-mcp-skills", + "explore-repo", + "fix-dependabot", + "get-context", + "handle-notifications", + "manage-labels", + "manage-project", + "manage-repo", + "manage-sub-issues", + "merge-pr", + "prepare-release", + "research-vulnerability", + "review-pr", + "search-code", + "security-audit", + "self-review-pr", + "share-snippet", + "trace-history", + "triage-issues", + "trigger-workflow" + ] + }, + { + "check": "enumerated by model: create-pr", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: address-pr-feedback", + "pass": true, + "detail": [ + "address-pr-feedback", + "browse-discussions", + "contribute-oss", + "create-issue", + "create-pr", + "debug-ci", + "delegate-to-copilot", + "discover-github", + "discover-mcp-skills", + "explore-repo", + "fix-dependabot", + "get-context", + "handle-notifications", + "manage-labels", + "manage-project", + "manage-repo", + "manage-sub-issues", + "merge-pr", + "prepare-release", + "research-vulnerability", + "review-pr", + "search-code", + "security-audit", + "self-review-pr", + "share-snippet", + "trace-history", + "triage-issues", + "trigger-workflow" + ] + }, + { + "check": "enumerated by model: address-pr-feedback", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "loaded: create-issue", + "pass": true, + "detail": { + "loaded": [ + "create-issue" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/results/uri-load.20260827-190213.json b/experiments/goose/results/uri-load.20260827-190213.json new file mode 100644 index 0000000..09f0f29 --- /dev/null +++ b/experiments/goose/results/uri-load.20260827-190213.json @@ -0,0 +1,240 @@ +{ + "scenario": "uri-load", + "timestamp": "2026-08-27T19:02:13", + "endpoint": "http://localhost:8082/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "address-pr-feedback", + "uri": "skill://github/address-pr-feedback/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "browse-discussions", + "uri": "skill://github/browse-discussions/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "contribute-oss", + "uri": "skill://github/contribute-oss/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "create-issue", + "uri": "skill://github/create-issue/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "create-pr", + "uri": "skill://github/create-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "debug-ci", + "uri": "skill://github/debug-ci/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "delegate-to-copilot", + "uri": "skill://github/delegate-to-copilot/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "discover-github", + "uri": "skill://github/discover-github/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "discover-mcp-skills", + "uri": "skill://github/discover-mcp-skills/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "explore-repo", + "uri": "skill://github/explore-repo/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "fix-dependabot", + "uri": "skill://github/fix-dependabot/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "get-context", + "uri": "skill://github/get-context/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "handle-notifications", + "uri": "skill://github/handle-notifications/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-labels", + "uri": "skill://github/manage-labels/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-project", + "uri": "skill://github/manage-project/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-repo", + "uri": "skill://github/manage-repo/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "manage-sub-issues", + "uri": "skill://github/manage-sub-issues/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "merge-pr", + "uri": "skill://github/merge-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "prepare-release", + "uri": "skill://github/prepare-release/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "research-vulnerability", + "uri": "skill://github/research-vulnerability/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "review-pr", + "uri": "skill://github/review-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "search-code", + "uri": "skill://github/search-code/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "security-audit", + "uri": "skill://github/security-audit/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "self-review-pr", + "uri": "skill://github/self-review-pr/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "share-snippet", + "uri": "skill://github/share-snippet/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "trace-history", + "uri": "skill://github/trace-history/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "triage-issues", + "uri": "skill://github/triage-issues/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "trigger-workflow", + "uri": "skill://github/trigger-workflow/SKILL.md", + "resources": 1, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "skill://github/create-pr/SKILL.md" + } + } + ], + "load_skill_calls": [ + "skill://github/create-pr/SKILL.md" + ], + "skills_loaded": [ + "skill://github/create-pr/SKILL.md" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260828_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-jip200wl\\wd\n L L goose is ready\n\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://github/create-pr/SKILL.md\n\nHere are the two pieces of information you asked for:\n\n- **First markdown heading line:** `# Create Pull Request`\n- **Origin server named in the loaded header:** `scenario_server`\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "resources_sized", + "pass": true, + "detail": { + "manifest_entries": 28, + "unsized": [] + } + }, + { + "check": "loaded: skill://github/create-pr/SKILL.md", + "pass": true, + "detail": { + "loaded": [ + "skill://github/create-pr/SKILL.md" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/run_scenario.py b/experiments/goose/run_scenario.py new file mode 100644 index 0000000..a9b8a7b --- /dev/null +++ b/experiments/goose/run_scenario.py @@ -0,0 +1,296 @@ +#!/usr/bin/env python3 +"""Run one SEP-2640 scenario against the goose CLI and a running server. + +Records discoverability (wire-level skills/list probe + whether the model +enumerated the skills from its instructions) and usage (tool calls, skills +loaded, resources read), then applies the scenario's `expect` checks. +See README.md for the scenario format. +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +import time +import urllib.request + +import yaml + +SKILLS_EXTENSION_ID = "io.modelcontextprotocol/skills" + + +def rpc(endpoint, token, method, params, req_id): + """POST one JSON-RPC request; tolerate plain-JSON or SSE-framed replies.""" + body = json.dumps( + {"jsonrpc": "2.0", "id": req_id, "method": method, "params": params} + ).encode() + headers = { + "Content-Type": "application/json", + "Accept": "application/json, text/event-stream", + } + if token: + headers["Authorization"] = f"Bearer {token}" + req = urllib.request.Request(endpoint, data=body, headers=headers) + raw = urllib.request.urlopen(req, timeout=30).read().decode("utf-8", "replace") + for line in raw.splitlines(): + if line.startswith("data: "): + raw = line[len("data: "):] + break + return json.loads(raw) + + +def probe_server(endpoint, token): + """Wire-level discoverability: capability declaration + skills/list names.""" + init = rpc(endpoint, token, "initialize", { + "protocolVersion": "2026-07-28", + "capabilities": {}, + "clientInfo": {"name": "sep2640-harness", "version": "0"}, + }, 1) + extensions = init["result"]["capabilities"].get("extensions", {}) or {} + declared = SKILLS_EXTENSION_ID in extensions + + skills, result_types = [], [] + if declared: + cursor, page = None, 0 + while page < 16: + page += 1 + params = {"cursor": cursor} if cursor else {} + result = rpc(endpoint, token, "skills/list", params, 1 + page)["result"] + result_types.append(result.get("resultType")) + skills.extend(result.get("skills", [])) + cursor = result.get("nextCursor") + if not cursor: + break + + def describe(s): + res = s.get("resources") + return { + "name": (s.get("frontmatter") or {}).get("name"), + "uri": s.get("uri"), + # "dynamic" marker, file count for a manifest, None if absent (invalid per SEP). + "resources": res if isinstance(res, str) else (len(res) if isinstance(res, list) else None), + # every manifest element carries an integer size (SEP baseline b405ba5) + "sized": all(isinstance(r.get("size"), int) for r in res) if isinstance(res, list) else None, + } + + return { + "extension_declared": declared, + "extension_settings": extensions.get(SKILLS_EXTENSION_ID), + "protocol_version": init["result"].get("protocolVersion"), + "result_type": result_types[0] if result_types else None, + "skills": [describe(s) for s in skills], + } + + +def write_goose_config(root, scenario, token): + provider = scenario.get("provider", "anthropic") + model = scenario.get("model", "claude-sonnet-4-6") + endpoint = scenario["mcp_server"]["endpoint"] + config = { + "providers": {provider: {"enabled": True, "model": model, "configured": True}}, + "active_provider": provider, + "extensions": { + "skills": { + "enabled": True, + "type": "platform", + "name": "skills", + "description": "", + }, + "scenario_server": { + "enabled": True, + "skills_enabled": True, + "type": "streamable_http", + "name": "scenario_server", + "uri": endpoint, + "description": "scenario MCP server", + "timeout": 300, + }, + }, + } + if token: + config["extensions"]["scenario_server"]["headers"] = { + "Authorization": f"Bearer {token}" + } + config_dir = os.path.join(root, "config") + os.makedirs(config_dir) + with open(os.path.join(config_dir, "config.yaml"), "w", encoding="utf-8") as f: + yaml.safe_dump(config, f) + + +# "▸ load_skill" or "▸ read_resource extensionmanager" (tool, then owning extension) +TOOL_LINE = re.compile(r"^\s*[▸>]\s+(\S+)(?:\s+\S+)?\s*$") +ARG_LINE = re.compile(r"^\s{2,}(\w+): (.*)$") + + +def parse_tool_calls(output): + """Extract (tool, {arg: value}) pairs from goose CLI output.""" + calls = [] + current = None + for line in output.splitlines(): + m = TOOL_LINE.match(line) + if m: + current = (m.group(1), {}) + calls.append(current) + continue + if current is not None: + m = ARG_LINE.match(line) + if m: + current[1][m.group(1)] = m.group(2).strip() + else: + current = None + return calls + + +def tool_results(root): + """Map tool_use id -> (tool name, result text) from goose's LLM request log. + + The CLI transcript shows only the call and its args; the request log holds + the tool_result blocks the model actually received. + """ + log_dir = os.path.join(root, "state", "logs") + names, results = {}, {} + for fn in sorted(os.listdir(log_dir)) if os.path.isdir(log_dir) else []: + if not fn.startswith("llm_request."): + continue + with open(os.path.join(log_dir, fn), encoding="utf-8") as f: + for line in f: + try: + req = json.loads(line) + except ValueError: + continue + for msg in req.get("input", req).get("messages", []): + content = msg.get("content") + if not isinstance(content, list): + continue + for block in content: + if block.get("type") == "tool_use": + names[block["id"]] = block["name"] + elif block.get("type") == "tool_result": + c = block.get("content") + if isinstance(c, list): + c = "".join(b.get("text", "") for b in c) + results[block["tool_use_id"]] = c or "" + return {tid: (names.get(tid), text) for tid, text in results.items()} + + +def run_goose(goose_bin, scenario, root, workdir): + env = dict(os.environ, GOOSE_PATH_ROOT=root) + proc = subprocess.run( + [goose_bin, "run", "-t", scenario["prompt"]], + cwd=workdir, + env=env, + capture_output=True, + timeout=600, + ) + return (proc.stdout + proc.stderr).decode("utf-8", "replace") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--scenario", required=True) + ap.add_argument("--goose", default=os.environ.get("GOOSE_BIN", "goose")) + args = ap.parse_args() + + with open(args.scenario, encoding="utf-8") as f: + scenario = yaml.safe_load(f) + expect = scenario.get("expect", {}) or {} + + bearer_cmd = scenario["mcp_server"].get("bearer_cmd") + token = "" + if bearer_cmd: + token = subprocess.run( + bearer_cmd, shell=True, capture_output=True, text=True, check=True + ).stdout.strip() + + server = probe_server(scenario["mcp_server"]["endpoint"], token) + wire_names = [s["name"] for s in server["skills"]] + + root = tempfile.mkdtemp(prefix="goose-scenario-") + workdir = os.path.join(root, "wd") + os.makedirs(workdir) + write_goose_config(root, scenario, token) + print(f"goose config root{' (contains bearer token)' if token else ''}: {root}") + + output = run_goose(args.goose, scenario, root, workdir) + + calls = parse_tool_calls(output) + tools_used = sorted({name for name, _ in calls}) + load_calls = [a["name"] for n, a in calls if n == "load_skill" and "name" in a] + load_results = [ + text for tid, (name, text) in tool_results(root).items() if name == "load_skill" + ] + # A load only counts when goose returned framed content, not an error: + # "# Loaded Skill:" for a SKILL.md, "# Loaded:" for a supporting file. + skills_loaded = [ + n for n, text in zip(load_calls, load_results) if text.startswith("# Loaded") + ] + load_failures = [ + (n, text[:200]) for n, text in zip(load_calls, load_results) + if not text.startswith("# Loaded") + ] + resources_read = [a["uri"] for n, a in calls if n == "read_resource" and "uri" in a] + + checks = [] + + def check(name, passed, detail): + checks.append({"check": name, "pass": bool(passed), "detail": detail}) + + if "extension_declared" in expect: + check("extension_declared", + server["extension_declared"] == expect["extension_declared"], + server["extension_settings"]) + if "resources_sized" in expect: + manifests = [s for s in server["skills"] if isinstance(s["resources"], int)] + check("resources_sized", + bool(manifests) and all(s["sized"] for s in manifests) == expect["resources_sized"], + {"manifest_entries": len(manifests), "unsized": [s["name"] for s in manifests if not s["sized"]]}) + for name in expect.get("discovered", []): + check(f"discovered on wire: {name}", name in wire_names, wire_names) + check(f"enumerated by model: {name}", name in output, "searched reply text") + for name in expect.get("loaded", []): + check(f"loaded: {name}", name in skills_loaded, + {"loaded": skills_loaded, "failed": load_failures}) + for name in expect.get("tools", []): + check(f"tool used: {name}", name in tools_used, tools_used) + for uri in expect.get("resources_read", []): + check(f"resource read: {uri}", uri in resources_read, resources_read) + + result = { + "scenario": scenario["name"], + "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"), + "endpoint": scenario["mcp_server"]["endpoint"], + "server": server, + "run": { + "tools_used": tools_used, + "tool_calls": [{"tool": n, "args": a} for n, a in calls], + "load_skill_calls": load_calls, + "skills_loaded": skills_loaded, + "load_failures": load_failures, + "resources_read": resources_read, + "output_tail": output[-2000:], + }, + "checks": checks, + "passed": all(c["pass"] for c in checks), + } + + results_dir = os.path.join(os.path.dirname(os.path.abspath(args.scenario)), "..", "results") + os.makedirs(results_dir, exist_ok=True) + out_path = os.path.join( + results_dir, f"{scenario['name']}.{time.strftime('%Y%m%d-%H%M%S')}.json" + ) + with open(out_path, "w", encoding="utf-8") as f: + json.dump(result, f, indent=2, ensure_ascii=False) + + for c in checks: + print(f"[{'PASS' if c['pass'] else 'FAIL'}] {c['check']}") + print(f"tools used: {tools_used}") + print(f"skills loaded: {skills_loaded}") + print(f"result: {out_path}") + sys.exit(0 if result["passed"] else 1) + + +if __name__ == "__main__": + main() diff --git a/experiments/goose/scenarios/demo-discovery-and-load.yaml b/experiments/goose/scenarios/demo-discovery-and-load.yaml new file mode 100644 index 0000000..f40f969 --- /dev/null +++ b/experiments/goose/scenarios/demo-discovery-and-load.yaml @@ -0,0 +1,13 @@ +name: demo-discovery-and-load +mcp_server: + endpoint: https://olaservo-skills-over-mcp-demo.hf.space/mcp +prompt: | + First: list every skill name that your system instructions say comes from a connected MCP server, one per line, exactly as written. + Second: call the load_skill tool with name 'tabletop-dice' and reply with the first markdown heading line of the loaded content. + Use no tools other than load_skill. +expect: + extension_declared: true + resources_sized: true + discovered: [tabletop-dice, mcp-glossary, release-notes-writer] + loaded: [tabletop-dice] + tools: [load_skill] diff --git a/experiments/goose/scenarios/demo-supporting-file.yaml b/experiments/goose/scenarios/demo-supporting-file.yaml new file mode 100644 index 0000000..d1886d4 --- /dev/null +++ b/experiments/goose/scenarios/demo-supporting-file.yaml @@ -0,0 +1,11 @@ +name: demo-supporting-file +mcp_server: + endpoint: https://olaservo-skills-over-mcp-demo.hf.space/mcp +prompt: | + Call the load_skill tool with name 'tabletop-dice'. The loaded content ends with a Supporting Files section that shows exactly how to load its supporting file; load it that way. + Reply with the first markdown heading line of the supporting file. + Use no tools other than load_skill. +expect: + extension_declared: true + loaded: [tabletop-dice, tabletop-dice/references/dice-notation.md] + tools: [load_skill] diff --git a/experiments/goose/scenarios/demo-unlisted-skill.yaml b/experiments/goose/scenarios/demo-unlisted-skill.yaml new file mode 100644 index 0000000..591e680 --- /dev/null +++ b/experiments/goose/scenarios/demo-unlisted-skill.yaml @@ -0,0 +1,11 @@ +name: demo-unlisted-skill +mcp_server: + endpoint: https://olaservo-skills-over-mcp-demo.hf.space/mcp +prompt: | + A connected MCP server's instructions point to a skill for coffee recommendations that is not in your skill list. + Call the load_skill tool with that skill's URI exactly as the instructions give it, and reply with the first markdown heading line of the loaded content. + Use no tools other than load_skill. +expect: + extension_declared: true + loaded: ["skill://secret-menu/SKILL.md"] + tools: [load_skill] diff --git a/experiments/goose/scenarios/discovery-and-load.yaml b/experiments/goose/scenarios/discovery-and-load.yaml new file mode 100644 index 0000000..6425edb --- /dev/null +++ b/experiments/goose/scenarios/discovery-and-load.yaml @@ -0,0 +1,14 @@ +name: discovery-and-load +mcp_server: + endpoint: http://localhost:8082/mcp + bearer_cmd: gh auth token +prompt: | + First: list every skill name that your system instructions say comes from a connected MCP server, one per line, exactly as written. + Second: call the load_skill tool with name 'create-issue' and reply with the first markdown heading line of the loaded content. + Use no tools other than load_skill. +expect: + extension_declared: true + resources_sized: true + discovered: [create-issue, create-pr, address-pr-feedback] + loaded: [create-issue] + tools: [load_skill] diff --git a/experiments/goose/scenarios/uri-load.yaml b/experiments/goose/scenarios/uri-load.yaml new file mode 100644 index 0000000..4e4792d --- /dev/null +++ b/experiments/goose/scenarios/uri-load.yaml @@ -0,0 +1,13 @@ +name: uri-load +mcp_server: + endpoint: http://localhost:8082/mcp + bearer_cmd: gh auth token +prompt: | + Call the load_skill tool with name 'skill://github/create-pr/SKILL.md' (the full URI, exactly as given). + Reply with the first markdown heading line of the loaded content and the origin server named in the loaded header. + Use no tools other than load_skill. +expect: + extension_declared: true + resources_sized: true + loaded: ["skill://github/create-pr/SKILL.md"] + tools: [load_skill] diff --git a/experiments/goose/sep-2640-goose-surfaces.svg b/experiments/goose/sep-2640-goose-surfaces.svg new file mode 100644 index 0000000..024f57f --- /dev/null +++ b/experiments/goose/sep-2640-goose-surfaces.svg @@ -0,0 +1,122 @@ + + Skills, MCP skills, resources, and tools in goose (skills-sep-2640-port) + + + + + + + + + + + + Skills, MCP-served skills, MCP resources, and MCP tools in goose (skills-sep-2640-port) + + + + Model + skill list in system prompt: • git-helper — local skill • deploy (github) — MCP skill, origin-tagged, gated by per-server skills_enabled + tools: load_skill · read_resource · github__create_issue + + + + goose host + + + + load_skill + one tool, routes by origin + FS registry + per-server MCP skill cache + + + + read_resource + any resource, + any connected server + + + + github__create_issue + prefixed MCP tool, + pass-through + + + + verification gate + sha256 + size per file + frontmatter == entry + unlisted file → refuse + mismatch → skills/get, retry once + + + + load_skill("git-helper") + + load_skill("deploy" | URI) + + read_resource(server, uri) + + github__create_issue(…) + + + + read SKILL.md + from disk + + + + + resources/read, + origin-bound + + + + "dynamic" entry: + no manifest — + skip gate; files from + directory/read walk + or resources/list; + reads unverified + + + + resources/read — + raw content, no verify, + no skill activation + + + + tools/call + + + + startup: + walk skill dirs + → prompt list + + + + on connect: + skills/list + → cache → + prompt list + + + + Local filesystem + ~/.goose/skills · project · plugins + git-helper/SKILL.md + + references/… + + + + MCP server — declares io.modelcontextprotocol/skills (directoryRead?) + skills/list · skills/get · resources/read · resources/directory/read · tools/call + skill://deploy/SKILL.md + supporting files, served as resources + entry: {uri, frontmatter, resources: [{uri, digest, size}] | "dynamic"} + + + Both load_skill origins return the same framing — “# Loaded Skill: deploy (mcp skill from github)” — so origin stays visible to the model. + solid — model-initiated call  ·  dashed — discovery/metadata  ·  blue — what SEP-2640 adds over plain resources + diff --git a/experiments/sep-2640-github-mcp-server-surfaces.svg b/experiments/sep-2640-github-mcp-server-surfaces.svg new file mode 100644 index 0000000..9ff3456 --- /dev/null +++ b/experiments/sep-2640-github-mcp-server-surfaces.svg @@ -0,0 +1,96 @@ + + + + + + + + + + + + Agent host (Claude Code, Copilot, gemini-cli, …) + builds one skill registry, loads SKILL.md into model context + + + + regular skills: + scan checkout, + read files + from disk + + + + MCP (initialize, JSON-RPC) + + + + github-mcp-server — feature/agent-skills-v2 + declares capabilities.extensions[“io.modelcontextprotocol/skills”] = { directoryRead: true } + + + + + Tools + tools/list · tools/call + GitHub tools + issues, PRs, actions, … + + list_repo_skills + finds SKILL.md dirs in any + repo, returns skill:// URIs + the model hands to + resources/read or skills/get + + + + + Resources + resources/read · templates + repo://… content + branches, commits, tags, PRs + + skill://github/<name>/… + bundled skill files (embedded) + + skill://{owner}/{repo}/ + {skill}/{file} + repo-hosted skill files + + + + + Skills extension (SEP-2640) + skills/list + bundled catalog: frontmatter + + {uri, digest, size} per file + + skills/get + entry by URI — bundled or + repo-hosted, listed or not + + resources/directory/read + ls for any skill directory, + both namespaces + + + + skills/ directory in this repo + 28 regular Agent Skills — plain SKILL.md directories, + readable by any agent that scans the checkout; + the same files are embedded into the binary and + served over MCP as skill://github/<name>/… + + + Any GitHub repository + skills/*/SKILL.md · skills/{ns}/*/SKILL.md · + plugins/*/skills/*/SKILL.md · */SKILL.md + never in skills/list (unenumerable catalog) — + reached by URI via skills/get and resources/read + + + + go:embed (build time) + + GitHub API (on demand) +