Add initial experiments from latest fork implementations of Goose (host application) and GitHub MCP server

experiments/README.md

@@ -0,0 +1,17 @@
+# Experiments
+
+Scenario harnesses for exercising SEP-2640 v1 (`skills/list` + `skills/get` + `resources/directory/read`) against real hosts and servers. This tree was restarted from scratch for the v1 protocol methods; the earlier multi-host harnesses (fast-agent, hermes, HF demos) built against the index.json draft live on the `feature/resource-sep-early-findings` branch for reference.
+
+Current harnesses:
+
+- [`goose/`](goose/) — drives the goose CLI (branch `skills-sep-2640-port` of https://github.com/olaservo/goose) against a running SEP-2640 server.
+
+Reference server for all harnesses: [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `feature/agent-skills-v2` of https://github.com/olaservo/github-mcp-server.
+
+![Skill surfaces of github-mcp-server on feature/agent-skills-v2](sep-2640-github-mcp-server-surfaces.svg)
+
+The 28 bundled skills are plain `SKILL.md` directories in the checkout, embedded at build time and served as `skill://github/<name>/…`; `skills/list` enumerates only these. Skills in any GitHub repository are reachable by URI (`skill://{owner}/{repo}/{skill}/{file}`) through `skills/get`, `resources/read`, and `resources/directory/read`, with `list_repo_skills` as the tool that finds them, but they never appear in `skills/list`.
+
+Second server: [skills-over-mcp-demo](https://github.com/olaservo/skills-over-mcp-demo), a small SEP-2640 server on the v2 TypeScript SDK, live and unauthenticated at `https://olaservo-skills-over-mcp-demo.hf.space/mcp`. It adds what the GitHub server lacks: an unlisted skill reachable only through the server's `instructions` pointer and `skills/get`, supporting files (so the digest and size gate is exercised), and a multi-segment skill path. Free Spaces sleep when idle; the harness's wire probe wakes it before goose connects.
+
+Conventions shared by all harnesses: the runner never spawns the MCP server — it connects to an already-running server at the scenario's `mcp_server.endpoint` — and the scenario file is always an explicit `--scenario <path>` argument.

experiments/goose/README.md

@@ -0,0 +1,48 @@
+# goose scenario harness
+
+Runs one scenario against the goose CLI and a running SEP-2640 server, and records two things: what was **discoverable** (the server's declared capability and `skills/list` names, probed directly on the wire, plus whether the model could enumerate the skills from its own instructions) and what was **used** (every tool call goose made, which skills were loaded, which resources were read).
+
+## Surfaces under test
+
+![Skills, MCP-served skills, MCP resources, and MCP tools in goose](sep-2640-goose-surfaces.svg)
+
+How goose (`skills-sep-2640-port`) exposes SEP-2640 to the model: local and MCP-served skills share one prompt list and one `load_skill` tool that routes by origin; MCP loads pass through the verification gate (sha256 + size per file, frontmatter identity, unlisted-file refusal, one `skills/get` retry); `read_resource` reads raw content from any server with no verification or skill activation; prefixed MCP tools pass straight through to `tools/call`. `"dynamic"` entries skip the gate.
+
+## Prerequisites
+
+- A goose build with skills-over-MCP support: branch `skills-sep-2640-port` of https://github.com/olaservo/goose (`cargo build -p goose-cli`). Point the harness at the binary with `--goose` or `GOOSE_BIN`. Use `fdf9ae6c` or later: the 2026-08-21/24 upstream merges left `OAuthStepUpClient` without `skills_list`/`skills_get`/`directory_read` forwarding, so every streamable HTTP server reported `Transport closed`.
+- A running SEP-2640 server. Reference: draft PR [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `olaservo:feature/agent-skills-v2` (`09ea2f9f` or later for the b405ba5 `size`/`"dynamic"` contract) — `go build -o github-mcp-server-skills.exe ./cmd/github-mcp-server`, then `./github-mcp-server-skills.exe http --port 8082`. The harness never starts the server; it connects to the scenario's `mcp_server.endpoint`.
+- Auth: when the scenario sets `mcp_server.bearer_cmd`, it is run once and its output sent as a `Authorization: Bearer …` header by both the wire probe and goose. Omit it for unauthenticated servers.
+- An LLM provider goose can use. The harness writes an isolated goose config (nothing in your real config is touched); provider credentials come from the system keyring or provider env vars as usual. Defaults: `anthropic` / `claude-sonnet-4-6`, overridable per scenario.
+
+Scenarios named `demo-*` run against the public demo Space instead (see [`../README.md`](../README.md)); they need no server or auth setup. Observed 2026-08-25: the live Space negotiates `2025-11-25` and its `skills/list` carries no `ttlMs`/`cacheScope`, unlike the demo repo's README claims for its current commit; the deployment may be older than the repo.
+
+## Run
+
+```
+uv run --with pyyaml run_scenario.py --scenario scenarios/discovery-and-load.yaml
+```
+
+The result JSON lands in `results/`, the isolated goose config root in a temp dir (path printed; it contains the bearer token when one was used, delete when done). Exit code is 0 only if every check passed.
+
+## Scenario format
+
+```yaml
+name: discovery-and-load
+mcp_server:
+  endpoint: http://localhost:8082/mcp
+  bearer_cmd: gh auth token        # optional; omit for unauthenticated servers
+provider: anthropic                # optional
+model: claude-sonnet-4-6           # optional
+prompt: |
+  ...single user turn given to goose...
+expect:
+  extension_declared: true         # server declares io.modelcontextprotocol/skills
+  resources_sized: true            # every manifest element in skills/list carries an integer size
+  discovered: [create-issue]       # ⊆ wire skills/list names AND each appears in the reply text
+  loaded: [create-issue]           # ⊆ load_skill calls whose result was framed content (`# Loaded Skill:` / `# Loaded:`), not an error
+  tools: [load_skill]              # optional, ⊆ tool names goose called
+  resources_read: [skill://…]      # optional, ⊆ the `uri` args of read_resource calls
+```
+
+Every `expect` key is optional; omitted keys are recorded but not asserted. The harness grades nothing else — it logs the full tool-call trace and the reply tail so a human can read what actually happened.

experiments/goose/results/demo-discovery-and-load.20260825-205208.json

@@ -0,0 +1,132 @@
+{
+  "scenario": "demo-discovery-and-load",
+  "timestamp": "2026-08-25T20:52:08",
+  "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp",
+  "server": {
+    "extension_declared": true,
+    "extension_settings": {
+      "directoryRead": true
+    },
+    "protocol_version": "2025-11-25",
+    "result_type": null,
+    "skills": [
+      {
+        "name": "tabletop-dice",
+        "uri": "skill://dice-roller/tabletop-dice/SKILL.md",
+        "resources": 2,
+        "sized": true
+      },
+      {
+        "name": "mcp-glossary",
+        "uri": "skill://mcp-glossary/SKILL.md",
+        "resources": 1,
+        "sized": true
+      },
+      {
+        "name": "release-notes-writer",
+        "uri": "skill://release-notes-writer/SKILL.md",
+        "resources": 2,
+        "sized": true
+      }
+    ]
+  },
+  "run": {
+    "tools_used": [
+      "load_skill"
+    ],
+    "tool_calls": [
+      {
+        "tool": "load_skill",
+        "args": {
+          "name": "tabletop-dice"
+        }
+      }
+    ],
+    "load_skill_calls": [
+      "tabletop-dice"
+    ],
+    "skills_loaded": [
+      "tabletop-dice"
+    ],
+    "load_failures": [],
+    "resources_read": [],
+    "output_tail": "\n    __( O)>  ● new session · anthropic claude-sonnet-4-6\n   \\____)    20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-pkf8edyi\\wd\n     L L     goose is ready\nHere are the skills listed as coming from connected MCP servers in my system instructions:\n\n- mcp-glossary (scenario_server)\n- release-notes-writer (scenario_server)\n- tabletop-dice (scenario_server)\n\n---\n\nNow loading the skill:\n  ────────────────────────────────────────\n  ▸ load_skill\n    name: tabletop-dice\n\nThe first markdown heading line of the loaded content is:\n\n**`# Tabletop Dice`**\n"
+  },
+  "checks": [
+    {
+      "check": "extension_declared",
+      "pass": true,
+      "detail": {
+        "directoryRead": true
+      }
+    },
+    {
+      "check": "resources_sized",
+      "pass": true,
+      "detail": {
+        "manifest_entries": 3,
+        "unsized": []
+      }
+    },
+    {
+      "check": "discovered on wire: tabletop-dice",
+      "pass": true,
+      "detail": [
+        "tabletop-dice",
+        "mcp-glossary",
+        "release-notes-writer"
+      ]
+    },
+    {
+      "check": "enumerated by model: tabletop-dice",
+      "pass": true,
+      "detail": "searched reply text"
+    },
+    {
+      "check": "discovered on wire: mcp-glossary",
+      "pass": true,
+      "detail": [
+        "tabletop-dice",
+        "mcp-glossary",
+        "release-notes-writer"
+      ]
+    },
+    {
+      "check": "enumerated by model: mcp-glossary",
+      "pass": true,
+      "detail": "searched reply text"
+    },
+    {
+      "check": "discovered on wire: release-notes-writer",
+      "pass": true,
+      "detail": [
+        "tabletop-dice",
+        "mcp-glossary",
+        "release-notes-writer"
+      ]
+    },
+    {
+      "check": "enumerated by model: release-notes-writer",
+      "pass": true,
+      "detail": "searched reply text"
+    },
+    {
+      "check": "loaded: tabletop-dice",
+      "pass": true,
+      "detail": {
+        "loaded": [
+          "tabletop-dice"
+        ],
+        "failed": []
+      }
+    },
+    {
+      "check": "tool used: load_skill",
+      "pass": true,
+      "detail": [
+        "load_skill"
+      ]
+    }
+  ],
+  "passed": true
+}

experiments/goose/results/demo-supporting-file.20260825-205332.json

@@ -0,0 +1,102 @@
+{
+  "scenario": "demo-supporting-file",
+  "timestamp": "2026-08-25T20:53:32",
+  "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp",
+  "server": {
+    "extension_declared": true,
+    "extension_settings": {
+      "directoryRead": true
+    },
+    "protocol_version": "2025-11-25",
+    "result_type": null,
+    "skills": [
+      {
+        "name": "tabletop-dice",
+        "uri": "skill://dice-roller/tabletop-dice/SKILL.md",
+        "resources": 2,
+        "sized": true
+      },
+      {
+        "name": "mcp-glossary",
+        "uri": "skill://mcp-glossary/SKILL.md",
+        "resources": 1,
+        "sized": true
+      },
+      {
+        "name": "release-notes-writer",
+        "uri": "skill://release-notes-writer/SKILL.md",
+        "resources": 2,
+        "sized": true
+      }
+    ]
+  },
+  "run": {
+    "tools_used": [
+      "load_skill"
+    ],
+    "tool_calls": [
+      {
+        "tool": "load_skill",
+        "args": {
+          "name": "tabletop-dice"
+        }
+      },
+      {
+        "tool": "load_skill",
+        "args": {
+          "name": "tabletop-dice/references/dice-notation.md"
+        }
+      }
+    ],
+    "load_skill_calls": [
+      "tabletop-dice",
+      "tabletop-dice/references/dice-notation.md"
+    ],
+    "skills_loaded": [
+      "tabletop-dice",
+      "tabletop-dice/references/dice-notation.md"
+    ],
+    "load_failures": [],
+    "resources_read": [],
+    "output_tail": "\n    __( O)>  ● new session · anthropic claude-sonnet-4-6\n   \\____)    20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-eoa6k3on\\wd\n     L L     goose is ready\nI'll load the tabletop-dice skill first, then follow its instructions for loading the supporting file.\n  ────────────────────────────────────────\n  ▸ load_skill\n    name: tabletop-dice\n\nNow I'll load the supporting file as instructed:\n  ────────────────────────────────────────\n  ▸ load_skill\n    name: tabletop-dice/references/dice-notation.md\n\nThe first markdown heading line of the supporting file is:\n\n```\n# Dice Notation Reference\n```\n"
+  },
+  "checks": [
+    {
+      "check": "extension_declared",
+      "pass": true,
+      "detail": {
+        "directoryRead": true
+      }
+    },
+    {
+      "check": "loaded: tabletop-dice",
+      "pass": true,
+      "detail": {
+        "loaded": [
+          "tabletop-dice",
+          "tabletop-dice/references/dice-notation.md"
+        ],
+        "failed": []
+      }
+    },
+    {
+      "check": "loaded: tabletop-dice/references/dice-notation.md",
+      "pass": true,
+      "detail": {
+        "loaded": [
+          "tabletop-dice",
+          "tabletop-dice/references/dice-notation.md"
+        ],
+        "failed": []
+      }
+    },
+    {
+      "check": "tool used: load_skill",
+      "pass": true,
+      "detail": [
+        "load_skill"
+      ]
+    }
+  ],
+  "passed": true
+}

experiments/goose/results/demo-unlisted-skill.20260825-205215.json

@@ -0,0 +1,82 @@
+{
+  "scenario": "demo-unlisted-skill",
+  "timestamp": "2026-08-25T20:52:15",
+  "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp",
+  "server": {
+    "extension_declared": true,
+    "extension_settings": {
+      "directoryRead": true
+    },
+    "protocol_version": "2025-11-25",
+    "result_type": null,
+    "skills": [
+      {
+        "name": "tabletop-dice",
+        "uri": "skill://dice-roller/tabletop-dice/SKILL.md",
+        "resources": 2,
+        "sized": true
+      },
+      {
+        "name": "mcp-glossary",
+        "uri": "skill://mcp-glossary/SKILL.md",
+        "resources": 1,
+        "sized": true
+      },
+      {
+        "name": "release-notes-writer",
+        "uri": "skill://release-notes-writer/SKILL.md",
+        "resources": 2,
+        "sized": true
+      }
+    ]
+  },
+  "run": {
+    "tools_used": [
+      "load_skill"
+    ],
+    "tool_calls": [
+      {
+        "tool": "load_skill",
+        "args": {
+          "name": "skill://secret-menu/SKILL.md"
+        }
+      }
+    ],
+    "load_skill_calls": [
+      "skill://secret-menu/SKILL.md"
+    ],
+    "skills_loaded": [
+      "skill://secret-menu/SKILL.md"
+    ],
+    "load_failures": [],
+    "resources_read": [],
+    "output_tail": "\n    __( O)>  ● new session · anthropic claude-sonnet-4-6\n   \\____)    20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-4l0b61r8\\wd\n     L L     goose is ready\n\n  ────────────────────────────────────────\n  ▸ load_skill\n    name: skill://secret-menu/SKILL.md\n\nThe first markdown heading line of the loaded content is:\n\n```\n# Secret Menu\n```\n"
+  },
+  "checks": [
+    {
+      "check": "extension_declared",
+      "pass": true,
+      "detail": {
+        "directoryRead": true
+      }
+    },
+    {
+      "check": "loaded: skill://secret-menu/SKILL.md",
+      "pass": true,
+      "detail": {
+        "loaded": [
+          "skill://secret-menu/SKILL.md"
+        ],
+        "failed": []
+      }
+    },
+    {
+      "check": "tool used: load_skill",
+      "pass": true,
+      "detail": [
+        "load_skill"
+      ]
+    }
+  ],
+  "passed": true
+}