From 1e2b3cf90241ac3782f25108a9434a58b4d6f57d Mon Sep 17 00:00:00 2001 From: olaservo Date: Sun, 16 Aug 2026 18:47:18 -0700 Subject: [PATCH 1/6] feat(experiments): goose scenario harness for SEP-2640 v1 Clean-slate restart of the experiments tree, goose host only. One single-file runner records wire-level discoverability (capability declaration + skills/list probe), model-level discoverability (skill names enumerated from instructions), and usage (tool calls, skills loaded, resources read), with optional expect checks. Two scenarios (name load, URI load) pass against github/github-mcp-server#3046 with goose branch skills-sep-2640-port; first results committed. Earlier multi-host harnesses remain on feature/resource-sep-early-findings. Co-Authored-By: Claude Fable 5 Signed-off-by: olaservo --- experiments/README.md | 9 + experiments/goose/README.md | 38 +++ .../discovery-and-load.20260816-184651.json | 312 ++++++++++++++++++ .../results/uri-load.20260816-184703.json | 195 +++++++++++ experiments/goose/run_scenario.py | 231 +++++++++++++ .../goose/scenarios/discovery-and-load.yaml | 12 + experiments/goose/scenarios/uri-load.yaml | 11 + 7 files changed, 808 insertions(+) create mode 100644 experiments/README.md create mode 100644 experiments/goose/README.md create mode 100644 experiments/goose/results/discovery-and-load.20260816-184651.json create mode 100644 experiments/goose/results/uri-load.20260816-184703.json create mode 100644 experiments/goose/run_scenario.py create mode 100644 experiments/goose/scenarios/discovery-and-load.yaml create mode 100644 experiments/goose/scenarios/uri-load.yaml diff --git a/experiments/README.md b/experiments/README.md new file mode 100644 index 0000000..1fb2ba4 --- /dev/null +++ b/experiments/README.md @@ -0,0 +1,9 @@ +# Experiments + +Scenario harnesses for exercising SEP-2640 v1 (`skills/list` + `skills/get` + `resources/directory/read`) against real hosts and servers. This tree was restarted from scratch for the v1 protocol methods; the earlier multi-host harnesses (fast-agent, hermes, HF demos) built against the index.json draft live on the `feature/resource-sep-early-findings` branch for reference. + +Current harnesses: + +- [`goose/`](goose/) — drives the goose CLI (branch `skills-sep-2640-port` of `olaservo/goose`) against a running SEP-2640 server. + +Conventions shared by all harnesses: the runner never spawns the MCP server — it connects to an already-running server at the scenario's `mcp_server.endpoint` — and the scenario file is always an explicit `--scenario ` argument. diff --git a/experiments/goose/README.md b/experiments/goose/README.md new file mode 100644 index 0000000..de59048 --- /dev/null +++ b/experiments/goose/README.md @@ -0,0 +1,38 @@ +# goose scenario harness + +Runs one scenario against the goose CLI and a running SEP-2640 server, and records two things: what was **discoverable** (the server's declared capability and `skills/list` names, probed directly on the wire, plus whether the model could enumerate the skills from its own instructions) and what was **used** (every tool call goose made, which skills were loaded, which resources were read). + +## Prerequisites + +- A goose build with skills-over-MCP support: branch `skills-sep-2640-port` of `olaservo/goose` (`cargo build -p goose-cli`). Point the harness at the binary with `--goose` or `GOOSE_BIN`. +- A running SEP-2640 server. Reference: draft PR [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `olaservo:feature/agent-skills-v2` — `go build -o github-mcp-server-skills.exe ./cmd/github-mcp-server`, then `./github-mcp-server-skills.exe http --port 8082`. The harness never starts the server; it connects to the scenario's `mcp_server.endpoint`. +- Auth: the scenario's `mcp_server.bearer_cmd` (default `gh auth token`) is run once and sent as a `Authorization: Bearer …` header by both the wire probe and goose. +- An LLM provider goose can use. The harness writes an isolated goose config (nothing in your real config is touched); provider credentials come from the system keyring or provider env vars as usual. Defaults: `anthropic` / `claude-sonnet-4-6`, overridable per scenario. + +## Run + +``` +uv run --with pyyaml run_scenario.py --scenario scenarios/discovery-and-load.yaml +``` + +The result JSON lands in `results/`, the isolated goose config root in a temp dir (path printed; it contains the bearer token, delete when done). Exit code is 0 only if every check passed. + +## Scenario format + +```yaml +name: discovery-and-load +mcp_server: + endpoint: http://localhost:8082/mcp + bearer_cmd: gh auth token # optional, this is the default +provider: anthropic # optional +model: claude-sonnet-4-6 # optional +prompt: | + ...single user turn given to goose... +expect: + extension_declared: true # server declares io.modelcontextprotocol/skills + discovered: [create-issue] # ⊆ wire skills/list names AND each appears in the reply text + loaded: [create-issue] # ⊆ the `name` args of load_skill calls goose made + tools: [load_skill] # optional, ⊆ tool names goose called +``` + +Every `expect` key is optional; omitted keys are recorded but not asserted. The harness grades nothing else — it logs the full tool-call trace and the reply tail so a human can read what actually happened. diff --git a/experiments/goose/results/discovery-and-load.20260816-184651.json b/experiments/goose/results/discovery-and-load.20260816-184651.json new file mode 100644 index 0000000..8c24657 --- /dev/null +++ b/experiments/goose/results/discovery-and-load.20260816-184651.json @@ -0,0 +1,312 @@ +{ + "scenario": "discovery-and-load", + "timestamp": "2026-08-16T18:46:51", + "endpoint": "http://localhost:8082/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "skills": [ + { + "name": "address-pr-feedback", + "uri": "skill://github/address-pr-feedback/SKILL.md", + "resource_count": 1 + }, + { + "name": "browse-discussions", + "uri": "skill://github/browse-discussions/SKILL.md", + "resource_count": 1 + }, + { + "name": "contribute-oss", + "uri": "skill://github/contribute-oss/SKILL.md", + "resource_count": 1 + }, + { + "name": "create-issue", + "uri": "skill://github/create-issue/SKILL.md", + "resource_count": 1 + }, + { + "name": "create-pr", + "uri": "skill://github/create-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "debug-ci", + "uri": "skill://github/debug-ci/SKILL.md", + "resource_count": 1 + }, + { + "name": "delegate-to-copilot", + "uri": "skill://github/delegate-to-copilot/SKILL.md", + "resource_count": 1 + }, + { + "name": "discover-github", + "uri": "skill://github/discover-github/SKILL.md", + "resource_count": 1 + }, + { + "name": "discover-mcp-skills", + "uri": "skill://github/discover-mcp-skills/SKILL.md", + "resource_count": 1 + }, + { + "name": "explore-repo", + "uri": "skill://github/explore-repo/SKILL.md", + "resource_count": 1 + }, + { + "name": "fix-dependabot", + "uri": "skill://github/fix-dependabot/SKILL.md", + "resource_count": 1 + }, + { + "name": "get-context", + "uri": "skill://github/get-context/SKILL.md", + "resource_count": 1 + }, + { + "name": "handle-notifications", + "uri": "skill://github/handle-notifications/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-labels", + "uri": "skill://github/manage-labels/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-project", + "uri": "skill://github/manage-project/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-repo", + "uri": "skill://github/manage-repo/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-sub-issues", + "uri": "skill://github/manage-sub-issues/SKILL.md", + "resource_count": 1 + }, + { + "name": "merge-pr", + "uri": "skill://github/merge-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "prepare-release", + "uri": "skill://github/prepare-release/SKILL.md", + "resource_count": 1 + }, + { + "name": "research-vulnerability", + "uri": "skill://github/research-vulnerability/SKILL.md", + "resource_count": 1 + }, + { + "name": "review-pr", + "uri": "skill://github/review-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "search-code", + "uri": "skill://github/search-code/SKILL.md", + "resource_count": 1 + }, + { + "name": "security-audit", + "uri": "skill://github/security-audit/SKILL.md", + "resource_count": 1 + }, + { + "name": "self-review-pr", + "uri": "skill://github/self-review-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "share-snippet", + "uri": "skill://github/share-snippet/SKILL.md", + "resource_count": 1 + }, + { + "name": "trace-history", + "uri": "skill://github/trace-history/SKILL.md", + "resource_count": 1 + }, + { + "name": "triage-issues", + "uri": "skill://github/triage-issues/SKILL.md", + "resource_count": 1 + }, + { + "name": "trigger-workflow", + "uri": "skill://github/trigger-workflow/SKILL.md", + "resource_count": 1 + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "create-issue" + } + } + ], + "skills_loaded": [ + "create-issue" + ], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260817_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-o0hxqqe5\\wd\n L L goose is ready\nI'll answer the first part from my system instructions, then call `load_skill` for the second part.\n ────────────────────────────────────────\n ▸ load_skill\n name: create-issue\n\n---\n\n**First — MCP server skills, exactly as written in system instructions:**\n\n- address-pr-feedback (scenario_server)\n- browse-discussions (scenario_server)\n- contribute-oss (scenario_server)\n- create-issue (scenario_server)\n- create-pr (scenario_server)\n- debug-ci (scenario_server)\n- delegate-to-copilot (scenario_server)\n- discover-github (scenario_server)\n- discover-mcp-skills (scenario_server)\n- explore-repo (scenario_server)\n- fix-dependabot (scenario_server)\n- get-context (scenario_server)\n- handle-notifications (scenario_server)\n- manage-labels (scenario_server)\n- manage-project (scenario_server)\n- manage-repo (scenario_server)\n- manage-sub-issues (scenario_server)\n- merge-pr (scenario_server)\n- prepare-release (scenario_server)\n- research-vulnerability (scenario_server)\n- review-pr (scenario_server)\n- search-code (scenario_server)\n- security-audit (scenario_server)\n- self-review-pr (scenario_server)\n- share-snippet (scenario_server)\n- trace-history (scenario_server)\n- triage-issues (scenario_server)\n- trigger-workflow (scenario_server)\n\n---\n\n**Second — first markdown heading line of the loaded `create-issue` skill:**\n\n`# Create Issue`\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "discovered on wire: create-issue", + "pass": true, + "detail": [ + "address-pr-feedback", + "browse-discussions", + "contribute-oss", + "create-issue", + "create-pr", + "debug-ci", + "delegate-to-copilot", + "discover-github", + "discover-mcp-skills", + "explore-repo", + "fix-dependabot", + "get-context", + "handle-notifications", + "manage-labels", + "manage-project", + "manage-repo", + "manage-sub-issues", + "merge-pr", + "prepare-release", + "research-vulnerability", + "review-pr", + "search-code", + "security-audit", + "self-review-pr", + "share-snippet", + "trace-history", + "triage-issues", + "trigger-workflow" + ] + }, + { + "check": "enumerated by model: create-issue", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: create-pr", + "pass": true, + "detail": [ + "address-pr-feedback", + "browse-discussions", + "contribute-oss", + "create-issue", + "create-pr", + "debug-ci", + "delegate-to-copilot", + "discover-github", + "discover-mcp-skills", + "explore-repo", + "fix-dependabot", + "get-context", + "handle-notifications", + "manage-labels", + "manage-project", + "manage-repo", + "manage-sub-issues", + "merge-pr", + "prepare-release", + "research-vulnerability", + "review-pr", + "search-code", + "security-audit", + "self-review-pr", + "share-snippet", + "trace-history", + "triage-issues", + "trigger-workflow" + ] + }, + { + "check": "enumerated by model: create-pr", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: address-pr-feedback", + "pass": true, + "detail": [ + "address-pr-feedback", + "browse-discussions", + "contribute-oss", + "create-issue", + "create-pr", + "debug-ci", + "delegate-to-copilot", + "discover-github", + "discover-mcp-skills", + "explore-repo", + "fix-dependabot", + "get-context", + "handle-notifications", + "manage-labels", + "manage-project", + "manage-repo", + "manage-sub-issues", + "merge-pr", + "prepare-release", + "research-vulnerability", + "review-pr", + "search-code", + "security-audit", + "self-review-pr", + "share-snippet", + "trace-history", + "triage-issues", + "trigger-workflow" + ] + }, + { + "check": "enumerated by model: address-pr-feedback", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "loaded: create-issue", + "pass": true, + "detail": [ + "create-issue" + ] + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/results/uri-load.20260816-184703.json b/experiments/goose/results/uri-load.20260816-184703.json new file mode 100644 index 0000000..5a4191b --- /dev/null +++ b/experiments/goose/results/uri-load.20260816-184703.json @@ -0,0 +1,195 @@ +{ + "scenario": "uri-load", + "timestamp": "2026-08-16T18:47:03", + "endpoint": "http://localhost:8082/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "skills": [ + { + "name": "address-pr-feedback", + "uri": "skill://github/address-pr-feedback/SKILL.md", + "resource_count": 1 + }, + { + "name": "browse-discussions", + "uri": "skill://github/browse-discussions/SKILL.md", + "resource_count": 1 + }, + { + "name": "contribute-oss", + "uri": "skill://github/contribute-oss/SKILL.md", + "resource_count": 1 + }, + { + "name": "create-issue", + "uri": "skill://github/create-issue/SKILL.md", + "resource_count": 1 + }, + { + "name": "create-pr", + "uri": "skill://github/create-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "debug-ci", + "uri": "skill://github/debug-ci/SKILL.md", + "resource_count": 1 + }, + { + "name": "delegate-to-copilot", + "uri": "skill://github/delegate-to-copilot/SKILL.md", + "resource_count": 1 + }, + { + "name": "discover-github", + "uri": "skill://github/discover-github/SKILL.md", + "resource_count": 1 + }, + { + "name": "discover-mcp-skills", + "uri": "skill://github/discover-mcp-skills/SKILL.md", + "resource_count": 1 + }, + { + "name": "explore-repo", + "uri": "skill://github/explore-repo/SKILL.md", + "resource_count": 1 + }, + { + "name": "fix-dependabot", + "uri": "skill://github/fix-dependabot/SKILL.md", + "resource_count": 1 + }, + { + "name": "get-context", + "uri": "skill://github/get-context/SKILL.md", + "resource_count": 1 + }, + { + "name": "handle-notifications", + "uri": "skill://github/handle-notifications/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-labels", + "uri": "skill://github/manage-labels/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-project", + "uri": "skill://github/manage-project/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-repo", + "uri": "skill://github/manage-repo/SKILL.md", + "resource_count": 1 + }, + { + "name": "manage-sub-issues", + "uri": "skill://github/manage-sub-issues/SKILL.md", + "resource_count": 1 + }, + { + "name": "merge-pr", + "uri": "skill://github/merge-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "prepare-release", + "uri": "skill://github/prepare-release/SKILL.md", + "resource_count": 1 + }, + { + "name": "research-vulnerability", + "uri": "skill://github/research-vulnerability/SKILL.md", + "resource_count": 1 + }, + { + "name": "review-pr", + "uri": "skill://github/review-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "search-code", + "uri": "skill://github/search-code/SKILL.md", + "resource_count": 1 + }, + { + "name": "security-audit", + "uri": "skill://github/security-audit/SKILL.md", + "resource_count": 1 + }, + { + "name": "self-review-pr", + "uri": "skill://github/self-review-pr/SKILL.md", + "resource_count": 1 + }, + { + "name": "share-snippet", + "uri": "skill://github/share-snippet/SKILL.md", + "resource_count": 1 + }, + { + "name": "trace-history", + "uri": "skill://github/trace-history/SKILL.md", + "resource_count": 1 + }, + { + "name": "triage-issues", + "uri": "skill://github/triage-issues/SKILL.md", + "resource_count": 1 + }, + { + "name": "trigger-workflow", + "uri": "skill://github/trigger-workflow/SKILL.md", + "resource_count": 1 + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "skill://github/create-pr/SKILL.md" + } + } + ], + "skills_loaded": [ + "skill://github/create-pr/SKILL.md" + ], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260817_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-wr4f_6sn\\wd\n L L goose is ready\nI'll load the skill using exactly the URI provided.\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://github/create-pr/SKILL.md\n\nHere are the two pieces of information you asked for:\n\n- **First markdown heading line:** `# Create Pull Request`\n- **Origin server named in the loaded header:** `scenario_server`\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "loaded: skill://github/create-pr/SKILL.md", + "pass": true, + "detail": [ + "skill://github/create-pr/SKILL.md" + ] + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/run_scenario.py b/experiments/goose/run_scenario.py new file mode 100644 index 0000000..8dd975d --- /dev/null +++ b/experiments/goose/run_scenario.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +"""Run one SEP-2640 scenario against the goose CLI and a running server. + +Records discoverability (wire-level skills/list probe + whether the model +enumerated the skills from its instructions) and usage (tool calls, skills +loaded, resources read), then applies the scenario's `expect` checks. +See README.md for the scenario format. +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +import time +import urllib.request + +import yaml + +SKILLS_EXTENSION_ID = "io.modelcontextprotocol/skills" + + +def rpc(endpoint, token, method, params, req_id): + """POST one JSON-RPC request; tolerate plain-JSON or SSE-framed replies.""" + body = json.dumps( + {"jsonrpc": "2.0", "id": req_id, "method": method, "params": params} + ).encode() + req = urllib.request.Request( + endpoint, + data=body, + headers={ + "Content-Type": "application/json", + "Accept": "application/json, text/event-stream", + "Authorization": f"Bearer {token}", + }, + ) + raw = urllib.request.urlopen(req, timeout=30).read().decode("utf-8", "replace") + for line in raw.splitlines(): + if line.startswith("data: "): + raw = line[len("data: "):] + break + return json.loads(raw) + + +def probe_server(endpoint, token): + """Wire-level discoverability: capability declaration + skills/list names.""" + init = rpc(endpoint, token, "initialize", { + "protocolVersion": "2025-06-18", + "capabilities": {}, + "clientInfo": {"name": "sep2640-harness", "version": "0"}, + }, 1) + extensions = init["result"]["capabilities"].get("extensions", {}) or {} + declared = SKILLS_EXTENSION_ID in extensions + + skills = [] + if declared: + cursor, page = None, 0 + while page < 16: + page += 1 + params = {"cursor": cursor} if cursor else {} + result = rpc(endpoint, token, "skills/list", params, 1 + page)["result"] + skills.extend(result.get("skills", [])) + cursor = result.get("nextCursor") + if not cursor: + break + + return { + "extension_declared": declared, + "extension_settings": extensions.get(SKILLS_EXTENSION_ID), + "skills": [ + { + "name": (s.get("frontmatter") or {}).get("name"), + "uri": s.get("uri"), + "resource_count": len(s["resources"]) if s.get("resources") is not None else None, + } + for s in skills + ], + } + + +def write_goose_config(root, scenario, token): + provider = scenario.get("provider", "anthropic") + model = scenario.get("model", "claude-sonnet-4-6") + endpoint = scenario["mcp_server"]["endpoint"] + config = { + "providers": {provider: {"enabled": True, "model": model, "configured": True}}, + "active_provider": provider, + "extensions": { + "skills": { + "enabled": True, + "type": "platform", + "name": "skills", + "description": "", + }, + "scenario_server": { + "enabled": True, + "skills_enabled": True, + "type": "streamable_http", + "name": "scenario_server", + "uri": endpoint, + "description": "scenario MCP server", + "timeout": 300, + "headers": {"Authorization": f"Bearer {token}"}, + }, + }, + } + config_dir = os.path.join(root, "config") + os.makedirs(config_dir) + with open(os.path.join(config_dir, "config.yaml"), "w", encoding="utf-8") as f: + yaml.safe_dump(config, f) + + +TOOL_LINE = re.compile(r"^\s*[▸>]\s+(\S+)\s*$") +ARG_LINE = re.compile(r"^\s{2,}(\w+): (.*)$") + + +def parse_tool_calls(output): + """Extract (tool, {arg: value}) pairs from goose CLI output.""" + calls = [] + current = None + for line in output.splitlines(): + m = TOOL_LINE.match(line) + if m: + current = (m.group(1), {}) + calls.append(current) + continue + if current is not None: + m = ARG_LINE.match(line) + if m: + current[1][m.group(1)] = m.group(2).strip() + else: + current = None + return calls + + +def run_goose(goose_bin, scenario, root, workdir): + env = dict(os.environ, GOOSE_PATH_ROOT=root) + proc = subprocess.run( + [goose_bin, "run", "-t", scenario["prompt"]], + cwd=workdir, + env=env, + capture_output=True, + timeout=600, + ) + return (proc.stdout + proc.stderr).decode("utf-8", "replace") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--scenario", required=True) + ap.add_argument("--goose", default=os.environ.get("GOOSE_BIN", "goose")) + args = ap.parse_args() + + with open(args.scenario, encoding="utf-8") as f: + scenario = yaml.safe_load(f) + expect = scenario.get("expect", {}) or {} + + bearer_cmd = scenario["mcp_server"].get("bearer_cmd", "gh auth token") + token = subprocess.run( + bearer_cmd, shell=True, capture_output=True, text=True, check=True + ).stdout.strip() + + server = probe_server(scenario["mcp_server"]["endpoint"], token) + wire_names = [s["name"] for s in server["skills"]] + + root = tempfile.mkdtemp(prefix="goose-scenario-") + workdir = os.path.join(root, "wd") + os.makedirs(workdir) + write_goose_config(root, scenario, token) + print(f"goose config root (contains bearer token): {root}") + + output = run_goose(args.goose, scenario, root, workdir) + + calls = parse_tool_calls(output) + tools_used = sorted({name for name, _ in calls}) + skills_loaded = [a["name"] for n, a in calls if n == "load_skill" and "name" in a] + resources_read = [a["uri"] for n, a in calls if n == "read_resource" and "uri" in a] + + checks = [] + + def check(name, passed, detail): + checks.append({"check": name, "pass": bool(passed), "detail": detail}) + + if "extension_declared" in expect: + check("extension_declared", + server["extension_declared"] == expect["extension_declared"], + server["extension_settings"]) + for name in expect.get("discovered", []): + check(f"discovered on wire: {name}", name in wire_names, wire_names) + check(f"enumerated by model: {name}", name in output, "searched reply text") + for name in expect.get("loaded", []): + check(f"loaded: {name}", name in skills_loaded, skills_loaded) + for name in expect.get("tools", []): + check(f"tool used: {name}", name in tools_used, tools_used) + + result = { + "scenario": scenario["name"], + "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"), + "endpoint": scenario["mcp_server"]["endpoint"], + "server": server, + "run": { + "tools_used": tools_used, + "tool_calls": [{"tool": n, "args": a} for n, a in calls], + "skills_loaded": skills_loaded, + "resources_read": resources_read, + "output_tail": output[-2000:], + }, + "checks": checks, + "passed": all(c["pass"] for c in checks), + } + + results_dir = os.path.join(os.path.dirname(os.path.abspath(args.scenario)), "..", "results") + os.makedirs(results_dir, exist_ok=True) + out_path = os.path.join( + results_dir, f"{scenario['name']}.{time.strftime('%Y%m%d-%H%M%S')}.json" + ) + with open(out_path, "w", encoding="utf-8") as f: + json.dump(result, f, indent=2, ensure_ascii=False) + + for c in checks: + print(f"[{'PASS' if c['pass'] else 'FAIL'}] {c['check']}") + print(f"tools used: {tools_used}") + print(f"skills loaded: {skills_loaded}") + print(f"result: {out_path}") + sys.exit(0 if result["passed"] else 1) + + +if __name__ == "__main__": + main() diff --git a/experiments/goose/scenarios/discovery-and-load.yaml b/experiments/goose/scenarios/discovery-and-load.yaml new file mode 100644 index 0000000..e941f8a --- /dev/null +++ b/experiments/goose/scenarios/discovery-and-load.yaml @@ -0,0 +1,12 @@ +name: discovery-and-load +mcp_server: + endpoint: http://localhost:8082/mcp +prompt: | + First: list every skill name that your system instructions say comes from a connected MCP server, one per line, exactly as written. + Second: call the load_skill tool with name 'create-issue' and reply with the first markdown heading line of the loaded content. + Use no tools other than load_skill. +expect: + extension_declared: true + discovered: [create-issue, create-pr, address-pr-feedback] + loaded: [create-issue] + tools: [load_skill] diff --git a/experiments/goose/scenarios/uri-load.yaml b/experiments/goose/scenarios/uri-load.yaml new file mode 100644 index 0000000..56e598a --- /dev/null +++ b/experiments/goose/scenarios/uri-load.yaml @@ -0,0 +1,11 @@ +name: uri-load +mcp_server: + endpoint: http://localhost:8082/mcp +prompt: | + Call the load_skill tool with name 'skill://github/create-pr/SKILL.md' (the full URI, exactly as given). + Reply with the first markdown heading line of the loaded content and the origin server named in the loaded header. + Use no tools other than load_skill. +expect: + extension_declared: true + loaded: ["skill://github/create-pr/SKILL.md"] + tools: [load_skill] From 1958cccb103725201e1bdc3a4b73121667e06663 Mon Sep 17 00:00:00 2001 From: olaservo Date: Tue, 25 Aug 2026 08:20:29 -0700 Subject: [PATCH 2/6] feat(experiments): rerun goose scenarios at SEP baseline b405ba5 Both sides moved to the b405ba5 resources contract ({uri, digest, size} or "dynamic", resultType on 2026-07-28+ sessions). The harness now records protocolVersion, resultType, and per-entry size on the wire probe, adds a resources_sized expectation, and grades `loaded` on the `# Loaded Skill:` frame in the tool result rather than on the call being made. First rerun failed: goose's upstream OAuthStepUpClient wrapper did not forward skills_list/skills_get/directory_read, so every streamable HTTP server reported "Transport closed". Fixed in goose at fdf9ae6c; both scenarios pass against github-mcp-server feature/agent-skills-v2 at 7d4ae593. Results from the older builds are dropped. Adds the goose and github-mcp-server surface diagrams, and a check-diagram skill that verifies a diagram's claims against the tip of the branch its README names. Claude-Session: https://claude.ai/code/session_017L3xguMgzVZQNrzu2AvKX2 Signed-off-by: olaservo --- .claude/skills/check-diagram/SKILL.md | 64 +++++++++ experiments/README.md | 8 +- experiments/goose/README.md | 13 +- ...> discovery-and-load.20260824-212736.json} | 111 +++++++++++----- ...703.json => uri-load.20260824-212742.json} | 111 +++++++++++----- experiments/goose/run_scenario.py | 82 ++++++++++-- .../goose/scenarios/discovery-and-load.yaml | 1 + experiments/goose/scenarios/uri-load.yaml | 1 + experiments/goose/sep-2640-goose-surfaces.svg | 122 ++++++++++++++++++ .../sep-2640-github-mcp-server-surfaces.svg | 96 ++++++++++++++ 10 files changed, 527 insertions(+), 82 deletions(-) create mode 100644 .claude/skills/check-diagram/SKILL.md rename experiments/goose/results/{discovery-and-load.20260816-184651.json => discovery-and-load.20260824-212736.json} (68%) rename experiments/goose/results/{uri-load.20260816-184703.json => uri-load.20260824-212742.json} (63%) create mode 100644 experiments/goose/sep-2640-goose-surfaces.svg create mode 100644 experiments/sep-2640-github-mcp-server-surfaces.svg diff --git a/.claude/skills/check-diagram/SKILL.md b/.claude/skills/check-diagram/SKILL.md new file mode 100644 index 0000000..ab71bf8 --- /dev/null +++ b/.claude/skills/check-diagram/SKILL.md @@ -0,0 +1,64 @@ +--- +name: check-diagram +description: Check that a skills-over-MCP surfaces diagram (SVG) matches the latest commit of the implementation branch it describes. Works for any client or server implementation. Use when asked whether a diagram is accurate, current, or needs updating after a branch moved. +--- + +# Check a diagram against the branch tip + +Input: a path to an SVG in this repo (`experiments/**/*.svg`). Optional: `owner/repo@branch` if the diagram does not name one. + +## 1. Identify the implementation + +Nothing about the implementation is fixed in this skill. Find it from the material itself: + +- the SVG text usually names the branch (grep it for `feature/`, `-port`, `branch`); +- the README that links the diagram (`grep -rl --include=*.md`) names the repo as a GitHub URL and the branch; +- if the diagram has no README link yet, the nearest README above it in `experiments/` names the reference server and host. + +If neither the diagram nor a README names a repo and branch, ask; do not guess from local clones. + +## 2. Get the branch tip, not a local checkout + +Local clones and worktrees go stale. Always resolve the remote tip first: + +``` +git ls-remote https://github.com// refs/heads/ +``` + +Then read code at that exact commit. Either fetch into an existing clone (`git fetch origin ` then `git show origin/:` and `git grep origin/ -- `), or shallow-clone into the scratchpad (`git clone --depth 1 --branch `). Never read from a working tree without confirming `git rev-parse HEAD` equals the `ls-remote` sha. + +Record the sha and its date in the report. + +## 3. Extract the diagram's claims + +Pull the text out of the SVG: + +``` +grep -o ']*>[^<]*' | sed 's/]*>//' +``` + +Turn every concrete statement into a checkable claim. Typical claim types: + +- protocol methods and capability declarations (`skills/list`, `capabilities.extensions["io.modelcontextprotocol/skills"] = { directoryRead: true }`) +- wire shapes and field names (`{uri, digest, size}`, `"dynamic"`, `resultType`) +- URI patterns (`skill://github//…`, `skill://{owner}/{repo}/{skill}/{file}`) +- tool names and what they route to (`load_skill`, `read_resource`, `list_repo_skills`) +- gates and behaviours (digest + size check, frontmatter identity, unlisted file refused, one `skills/get` retry, per-server `skills_enabled`) +- counts (`28 regular Agent Skills`) and the branch name itself + +Layout, colour, and wording that carries no fact are not claims. + +## 4. Verify each claim in the code at the tip + +For each claim find the code that implements it and cite `path:line` at the checked sha. Search, do not recall: method names in request handlers or dispatch, field names in the wire structs, URI prefixes as string constants, counts by listing the directory. A claim is one of: + +- **matches** — code at the tip does what the diagram says; cite the evidence +- **stale** — the tip does something different; say what, cite the evidence +- **missing** — the tip has a surface the diagram does not show and a reader would expect on it +- **unverifiable** — could not find evidence either way; say what was searched + +Then compare against the SEP baseline the branch claims to track (`docs/sep-draft-skills-extension.md` header pins the commit; the PR is authoritative per AGENTS.md). Flag any diagram claim that matches the code but no longer matches the spec, since the code will change next. + +## 5. Report + +Plain markdown, one table row per claim, then the branch sha/date and the SEP commit compared. Below the table, list the exact SVG text edits needed for each stale or missing item. Do not edit the SVG unless asked; the diagrams are hand-authored and the user updates them. diff --git a/experiments/README.md b/experiments/README.md index 1fb2ba4..36cba24 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -4,6 +4,12 @@ Scenario harnesses for exercising SEP-2640 v1 (`skills/list` + `skills/get` + `r Current harnesses: -- [`goose/`](goose/) — drives the goose CLI (branch `skills-sep-2640-port` of `olaservo/goose`) against a running SEP-2640 server. +- [`goose/`](goose/) — drives the goose CLI (branch `skills-sep-2640-port` of https://github.com/olaservo/goose) against a running SEP-2640 server. + +Reference server for all harnesses: [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `feature/agent-skills-v2` of https://github.com/olaservo/github-mcp-server. + +![Skill surfaces of github-mcp-server on feature/agent-skills-v2](sep-2640-github-mcp-server-surfaces.svg) + +The 28 bundled skills are plain `SKILL.md` directories in the checkout, embedded at build time and served as `skill://github//…`; `skills/list` enumerates only these. Skills in any GitHub repository are reachable by URI (`skill://{owner}/{repo}/{skill}/{file}`) through `skills/get`, `resources/read`, and `resources/directory/read`, with `list_repo_skills` as the tool that finds them, but they never appear in `skills/list`. Conventions shared by all harnesses: the runner never spawns the MCP server — it connects to an already-running server at the scenario's `mcp_server.endpoint` — and the scenario file is always an explicit `--scenario ` argument. diff --git a/experiments/goose/README.md b/experiments/goose/README.md index de59048..865b54b 100644 --- a/experiments/goose/README.md +++ b/experiments/goose/README.md @@ -2,10 +2,16 @@ Runs one scenario against the goose CLI and a running SEP-2640 server, and records two things: what was **discoverable** (the server's declared capability and `skills/list` names, probed directly on the wire, plus whether the model could enumerate the skills from its own instructions) and what was **used** (every tool call goose made, which skills were loaded, which resources were read). +## Surfaces under test + +![Skills, MCP-served skills, MCP resources, and MCP tools in goose](sep-2640-goose-surfaces.svg) + +How goose (`skills-sep-2640-port`) exposes SEP-2640 to the model: local and MCP-served skills share one prompt list and one `load_skill` tool that routes by origin; MCP loads pass through the verification gate (sha256 + size per file, frontmatter identity, unlisted-file refusal, one `skills/get` retry); `read_resource` reads raw content from any server with no verification or skill activation; prefixed MCP tools pass straight through to `tools/call`. `"dynamic"` entries skip the gate. + ## Prerequisites -- A goose build with skills-over-MCP support: branch `skills-sep-2640-port` of `olaservo/goose` (`cargo build -p goose-cli`). Point the harness at the binary with `--goose` or `GOOSE_BIN`. -- A running SEP-2640 server. Reference: draft PR [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `olaservo:feature/agent-skills-v2` — `go build -o github-mcp-server-skills.exe ./cmd/github-mcp-server`, then `./github-mcp-server-skills.exe http --port 8082`. The harness never starts the server; it connects to the scenario's `mcp_server.endpoint`. +- A goose build with skills-over-MCP support: branch `skills-sep-2640-port` of https://github.com/olaservo/goose (`cargo build -p goose-cli`). Point the harness at the binary with `--goose` or `GOOSE_BIN`. Use `fdf9ae6c` or later: the 2026-08-21/24 upstream merges left `OAuthStepUpClient` without `skills_list`/`skills_get`/`directory_read` forwarding, so every streamable HTTP server reported `Transport closed`. +- A running SEP-2640 server. Reference: draft PR [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `olaservo:feature/agent-skills-v2` (`09ea2f9f` or later for the b405ba5 `size`/`"dynamic"` contract) — `go build -o github-mcp-server-skills.exe ./cmd/github-mcp-server`, then `./github-mcp-server-skills.exe http --port 8082`. The harness never starts the server; it connects to the scenario's `mcp_server.endpoint`. - Auth: the scenario's `mcp_server.bearer_cmd` (default `gh auth token`) is run once and sent as a `Authorization: Bearer …` header by both the wire probe and goose. - An LLM provider goose can use. The harness writes an isolated goose config (nothing in your real config is touched); provider credentials come from the system keyring or provider env vars as usual. Defaults: `anthropic` / `claude-sonnet-4-6`, overridable per scenario. @@ -30,8 +36,9 @@ prompt: | ...single user turn given to goose... expect: extension_declared: true # server declares io.modelcontextprotocol/skills + resources_sized: true # every manifest element in skills/list carries an integer size discovered: [create-issue] # ⊆ wire skills/list names AND each appears in the reply text - loaded: [create-issue] # ⊆ the `name` args of load_skill calls goose made + loaded: [create-issue] # ⊆ load_skill calls whose result was framed skill content (`# Loaded Skill:`), not an error tools: [load_skill] # optional, ⊆ tool names goose called ``` diff --git a/experiments/goose/results/discovery-and-load.20260816-184651.json b/experiments/goose/results/discovery-and-load.20260824-212736.json similarity index 68% rename from experiments/goose/results/discovery-and-load.20260816-184651.json rename to experiments/goose/results/discovery-and-load.20260824-212736.json index 8c24657..2389b16 100644 --- a/experiments/goose/results/discovery-and-load.20260816-184651.json +++ b/experiments/goose/results/discovery-and-load.20260824-212736.json @@ -1,152 +1,182 @@ { "scenario": "discovery-and-load", - "timestamp": "2026-08-16T18:46:51", + "timestamp": "2026-08-24T21:27:36", "endpoint": "http://localhost:8082/mcp", "server": { "extension_declared": true, "extension_settings": { "directoryRead": true }, + "protocol_version": "2025-11-25", + "result_type": null, "skills": [ { "name": "address-pr-feedback", "uri": "skill://github/address-pr-feedback/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "browse-discussions", "uri": "skill://github/browse-discussions/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "contribute-oss", "uri": "skill://github/contribute-oss/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "create-issue", "uri": "skill://github/create-issue/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "create-pr", "uri": "skill://github/create-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "debug-ci", "uri": "skill://github/debug-ci/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "delegate-to-copilot", "uri": "skill://github/delegate-to-copilot/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "discover-github", "uri": "skill://github/discover-github/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "discover-mcp-skills", "uri": "skill://github/discover-mcp-skills/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "explore-repo", "uri": "skill://github/explore-repo/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "fix-dependabot", "uri": "skill://github/fix-dependabot/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "get-context", "uri": "skill://github/get-context/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "handle-notifications", "uri": "skill://github/handle-notifications/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-labels", "uri": "skill://github/manage-labels/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-project", "uri": "skill://github/manage-project/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-repo", "uri": "skill://github/manage-repo/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-sub-issues", "uri": "skill://github/manage-sub-issues/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "merge-pr", "uri": "skill://github/merge-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "prepare-release", "uri": "skill://github/prepare-release/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "research-vulnerability", "uri": "skill://github/research-vulnerability/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "review-pr", "uri": "skill://github/review-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "search-code", "uri": "skill://github/search-code/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "security-audit", "uri": "skill://github/security-audit/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "self-review-pr", "uri": "skill://github/self-review-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "share-snippet", "uri": "skill://github/share-snippet/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "trace-history", "uri": "skill://github/trace-history/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "triage-issues", "uri": "skill://github/triage-issues/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "trigger-workflow", "uri": "skill://github/trigger-workflow/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true } ] }, @@ -162,11 +192,15 @@ } } ], + "load_skill_calls": [ + "create-issue" + ], "skills_loaded": [ "create-issue" ], + "load_failures": [], "resources_read": [], - "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260817_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-o0hxqqe5\\wd\n L L goose is ready\nI'll answer the first part from my system instructions, then call `load_skill` for the second part.\n ────────────────────────────────────────\n ▸ load_skill\n name: create-issue\n\n---\n\n**First — MCP server skills, exactly as written in system instructions:**\n\n- address-pr-feedback (scenario_server)\n- browse-discussions (scenario_server)\n- contribute-oss (scenario_server)\n- create-issue (scenario_server)\n- create-pr (scenario_server)\n- debug-ci (scenario_server)\n- delegate-to-copilot (scenario_server)\n- discover-github (scenario_server)\n- discover-mcp-skills (scenario_server)\n- explore-repo (scenario_server)\n- fix-dependabot (scenario_server)\n- get-context (scenario_server)\n- handle-notifications (scenario_server)\n- manage-labels (scenario_server)\n- manage-project (scenario_server)\n- manage-repo (scenario_server)\n- manage-sub-issues (scenario_server)\n- merge-pr (scenario_server)\n- prepare-release (scenario_server)\n- research-vulnerability (scenario_server)\n- review-pr (scenario_server)\n- search-code (scenario_server)\n- security-audit (scenario_server)\n- self-review-pr (scenario_server)\n- share-snippet (scenario_server)\n- trace-history (scenario_server)\n- triage-issues (scenario_server)\n- trigger-workflow (scenario_server)\n\n---\n\n**Second — first markdown heading line of the loaded `create-issue` skill:**\n\n`# Create Issue`\n" + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260825_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-7g8x1h6y\\wd\n L L goose is ready\nHere are the skills from connected MCP servers, exactly as written in my system instructions:\n\n- address-pr-feedback (scenario_server)\n- browse-discussions (scenario_server)\n- contribute-oss (scenario_server)\n- create-issue (scenario_server)\n- create-pr (scenario_server)\n- debug-ci (scenario_server)\n- delegate-to-copilot (scenario_server)\n- discover-github (scenario_server)\n- discover-mcp-skills (scenario_server)\n- explore-repo (scenario_server)\n- fix-dependabot (scenario_server)\n- get-context (scenario_server)\n- handle-notifications (scenario_server)\n- manage-labels (scenario_server)\n- manage-project (scenario_server)\n- manage-repo (scenario_server)\n- manage-sub-issues (scenario_server)\n- merge-pr (scenario_server)\n- prepare-release (scenario_server)\n- research-vulnerability (scenario_server)\n- review-pr (scenario_server)\n- search-code (scenario_server)\n- security-audit (scenario_server)\n- self-review-pr (scenario_server)\n- share-snippet (scenario_server)\n- trace-history (scenario_server)\n- triage-issues (scenario_server)\n- trigger-workflow (scenario_server)\n\nNow loading the `create-issue` skill:\n ────────────────────────────────────────\n ▸ load_skill\n name: create-issue\n\nThe first markdown heading line of the loaded content is:\n\n**`# Create Issue`**\n" }, "checks": [ { @@ -176,6 +210,14 @@ "directoryRead": true } }, + { + "check": "resources_sized", + "pass": true, + "detail": { + "manifest_entries": 28, + "unsized": [] + } + }, { "check": "discovered on wire: create-issue", "pass": true, @@ -296,9 +338,12 @@ { "check": "loaded: create-issue", "pass": true, - "detail": [ - "create-issue" - ] + "detail": { + "loaded": [ + "create-issue" + ], + "failed": [] + } }, { "check": "tool used: load_skill", diff --git a/experiments/goose/results/uri-load.20260816-184703.json b/experiments/goose/results/uri-load.20260824-212742.json similarity index 63% rename from experiments/goose/results/uri-load.20260816-184703.json rename to experiments/goose/results/uri-load.20260824-212742.json index 5a4191b..64527f9 100644 --- a/experiments/goose/results/uri-load.20260816-184703.json +++ b/experiments/goose/results/uri-load.20260824-212742.json @@ -1,152 +1,182 @@ { "scenario": "uri-load", - "timestamp": "2026-08-16T18:47:03", + "timestamp": "2026-08-24T21:27:42", "endpoint": "http://localhost:8082/mcp", "server": { "extension_declared": true, "extension_settings": { "directoryRead": true }, + "protocol_version": "2025-11-25", + "result_type": null, "skills": [ { "name": "address-pr-feedback", "uri": "skill://github/address-pr-feedback/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "browse-discussions", "uri": "skill://github/browse-discussions/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "contribute-oss", "uri": "skill://github/contribute-oss/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "create-issue", "uri": "skill://github/create-issue/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "create-pr", "uri": "skill://github/create-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "debug-ci", "uri": "skill://github/debug-ci/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "delegate-to-copilot", "uri": "skill://github/delegate-to-copilot/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "discover-github", "uri": "skill://github/discover-github/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "discover-mcp-skills", "uri": "skill://github/discover-mcp-skills/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "explore-repo", "uri": "skill://github/explore-repo/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "fix-dependabot", "uri": "skill://github/fix-dependabot/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "get-context", "uri": "skill://github/get-context/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "handle-notifications", "uri": "skill://github/handle-notifications/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-labels", "uri": "skill://github/manage-labels/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-project", "uri": "skill://github/manage-project/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-repo", "uri": "skill://github/manage-repo/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "manage-sub-issues", "uri": "skill://github/manage-sub-issues/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "merge-pr", "uri": "skill://github/merge-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "prepare-release", "uri": "skill://github/prepare-release/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "research-vulnerability", "uri": "skill://github/research-vulnerability/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "review-pr", "uri": "skill://github/review-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "search-code", "uri": "skill://github/search-code/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "security-audit", "uri": "skill://github/security-audit/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "self-review-pr", "uri": "skill://github/self-review-pr/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "share-snippet", "uri": "skill://github/share-snippet/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "trace-history", "uri": "skill://github/trace-history/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "triage-issues", "uri": "skill://github/triage-issues/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true }, { "name": "trigger-workflow", "uri": "skill://github/trigger-workflow/SKILL.md", - "resource_count": 1 + "resources": 1, + "sized": true } ] }, @@ -162,11 +192,15 @@ } } ], + "load_skill_calls": [ + "skill://github/create-pr/SKILL.md" + ], "skills_loaded": [ "skill://github/create-pr/SKILL.md" ], + "load_failures": [], "resources_read": [], - "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260817_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-wr4f_6sn\\wd\n L L goose is ready\nI'll load the skill using exactly the URI provided.\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://github/create-pr/SKILL.md\n\nHere are the two pieces of information you asked for:\n\n- **First markdown heading line:** `# Create Pull Request`\n- **Origin server named in the loaded header:** `scenario_server`\n" + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260825_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-5bfr0zjv\\wd\n L L goose is ready\nI'll load the skill using the exact URI provided.\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://github/create-pr/SKILL.md\n\nHere are the two requested pieces of information:\n\n- **First markdown heading line:** `# Create Pull Request`\n- **Origin server named in the loaded header:** `scenario_server`\n" }, "checks": [ { @@ -176,12 +210,23 @@ "directoryRead": true } }, + { + "check": "resources_sized", + "pass": true, + "detail": { + "manifest_entries": 28, + "unsized": [] + } + }, { "check": "loaded: skill://github/create-pr/SKILL.md", "pass": true, - "detail": [ - "skill://github/create-pr/SKILL.md" - ] + "detail": { + "loaded": [ + "skill://github/create-pr/SKILL.md" + ], + "failed": [] + } }, { "check": "tool used: load_skill", diff --git a/experiments/goose/run_scenario.py b/experiments/goose/run_scenario.py index 8dd975d..9552b20 100644 --- a/experiments/goose/run_scenario.py +++ b/experiments/goose/run_scenario.py @@ -47,36 +47,43 @@ def rpc(endpoint, token, method, params, req_id): def probe_server(endpoint, token): """Wire-level discoverability: capability declaration + skills/list names.""" init = rpc(endpoint, token, "initialize", { - "protocolVersion": "2025-06-18", + "protocolVersion": "2026-07-28", "capabilities": {}, "clientInfo": {"name": "sep2640-harness", "version": "0"}, }, 1) extensions = init["result"]["capabilities"].get("extensions", {}) or {} declared = SKILLS_EXTENSION_ID in extensions - skills = [] + skills, result_types = [], [] if declared: cursor, page = None, 0 while page < 16: page += 1 params = {"cursor": cursor} if cursor else {} result = rpc(endpoint, token, "skills/list", params, 1 + page)["result"] + result_types.append(result.get("resultType")) skills.extend(result.get("skills", [])) cursor = result.get("nextCursor") if not cursor: break + def describe(s): + res = s.get("resources") + return { + "name": (s.get("frontmatter") or {}).get("name"), + "uri": s.get("uri"), + # "dynamic" marker, file count for a manifest, None if absent (invalid per SEP). + "resources": res if isinstance(res, str) else (len(res) if isinstance(res, list) else None), + # every manifest element carries an integer size (SEP baseline b405ba5) + "sized": all(isinstance(r.get("size"), int) for r in res) if isinstance(res, list) else None, + } + return { "extension_declared": declared, "extension_settings": extensions.get(SKILLS_EXTENSION_ID), - "skills": [ - { - "name": (s.get("frontmatter") or {}).get("name"), - "uri": s.get("uri"), - "resource_count": len(s["resources"]) if s.get("resources") is not None else None, - } - for s in skills - ], + "protocol_version": init["result"].get("protocolVersion"), + "result_type": result_types[0] if result_types else None, + "skills": [describe(s) for s in skills], } @@ -135,6 +142,38 @@ def parse_tool_calls(output): return calls +def tool_results(root): + """Map tool_use id -> (tool name, result text) from goose's LLM request log. + + The CLI transcript shows only the call and its args; the request log holds + the tool_result blocks the model actually received. + """ + log_dir = os.path.join(root, "state", "logs") + names, results = {}, {} + for fn in sorted(os.listdir(log_dir)) if os.path.isdir(log_dir) else []: + if not fn.startswith("llm_request."): + continue + with open(os.path.join(log_dir, fn), encoding="utf-8") as f: + for line in f: + try: + req = json.loads(line) + except ValueError: + continue + for msg in req.get("input", req).get("messages", []): + content = msg.get("content") + if not isinstance(content, list): + continue + for block in content: + if block.get("type") == "tool_use": + names[block["id"]] = block["name"] + elif block.get("type") == "tool_result": + c = block.get("content") + if isinstance(c, list): + c = "".join(b.get("text", "") for b in c) + results[block["tool_use_id"]] = c or "" + return {tid: (names.get(tid), text) for tid, text in results.items()} + + def run_goose(goose_bin, scenario, root, workdir): env = dict(os.environ, GOOSE_PATH_ROOT=root) proc = subprocess.run( @@ -175,7 +214,18 @@ def main(): calls = parse_tool_calls(output) tools_used = sorted({name for name, _ in calls}) - skills_loaded = [a["name"] for n, a in calls if n == "load_skill" and "name" in a] + load_calls = [a["name"] for n, a in calls if n == "load_skill" and "name" in a] + load_results = [ + text for tid, (name, text) in tool_results(root).items() if name == "load_skill" + ] + # A load only counts when goose returned framed skill content, not an error. + skills_loaded = [ + n for n, text in zip(load_calls, load_results) if text.startswith("# Loaded Skill:") + ] + load_failures = [ + (n, text[:200]) for n, text in zip(load_calls, load_results) + if not text.startswith("# Loaded Skill:") + ] resources_read = [a["uri"] for n, a in calls if n == "read_resource" and "uri" in a] checks = [] @@ -187,11 +237,17 @@ def check(name, passed, detail): check("extension_declared", server["extension_declared"] == expect["extension_declared"], server["extension_settings"]) + if "resources_sized" in expect: + manifests = [s for s in server["skills"] if isinstance(s["resources"], int)] + check("resources_sized", + bool(manifests) and all(s["sized"] for s in manifests) == expect["resources_sized"], + {"manifest_entries": len(manifests), "unsized": [s["name"] for s in manifests if not s["sized"]]}) for name in expect.get("discovered", []): check(f"discovered on wire: {name}", name in wire_names, wire_names) check(f"enumerated by model: {name}", name in output, "searched reply text") for name in expect.get("loaded", []): - check(f"loaded: {name}", name in skills_loaded, skills_loaded) + check(f"loaded: {name}", name in skills_loaded, + {"loaded": skills_loaded, "failed": load_failures}) for name in expect.get("tools", []): check(f"tool used: {name}", name in tools_used, tools_used) @@ -203,7 +259,9 @@ def check(name, passed, detail): "run": { "tools_used": tools_used, "tool_calls": [{"tool": n, "args": a} for n, a in calls], + "load_skill_calls": load_calls, "skills_loaded": skills_loaded, + "load_failures": load_failures, "resources_read": resources_read, "output_tail": output[-2000:], }, diff --git a/experiments/goose/scenarios/discovery-and-load.yaml b/experiments/goose/scenarios/discovery-and-load.yaml index e941f8a..18840dd 100644 --- a/experiments/goose/scenarios/discovery-and-load.yaml +++ b/experiments/goose/scenarios/discovery-and-load.yaml @@ -7,6 +7,7 @@ prompt: | Use no tools other than load_skill. expect: extension_declared: true + resources_sized: true discovered: [create-issue, create-pr, address-pr-feedback] loaded: [create-issue] tools: [load_skill] diff --git a/experiments/goose/scenarios/uri-load.yaml b/experiments/goose/scenarios/uri-load.yaml index 56e598a..136e306 100644 --- a/experiments/goose/scenarios/uri-load.yaml +++ b/experiments/goose/scenarios/uri-load.yaml @@ -7,5 +7,6 @@ prompt: | Use no tools other than load_skill. expect: extension_declared: true + resources_sized: true loaded: ["skill://github/create-pr/SKILL.md"] tools: [load_skill] diff --git a/experiments/goose/sep-2640-goose-surfaces.svg b/experiments/goose/sep-2640-goose-surfaces.svg new file mode 100644 index 0000000..024f57f --- /dev/null +++ b/experiments/goose/sep-2640-goose-surfaces.svg @@ -0,0 +1,122 @@ + + Skills, MCP skills, resources, and tools in goose (skills-sep-2640-port) + + + + + + + + + + + + Skills, MCP-served skills, MCP resources, and MCP tools in goose (skills-sep-2640-port) + + + + Model + skill list in system prompt: • git-helper — local skill • deploy (github) — MCP skill, origin-tagged, gated by per-server skills_enabled + tools: load_skill · read_resource · github__create_issue + + + + goose host + + + + load_skill + one tool, routes by origin + FS registry + per-server MCP skill cache + + + + read_resource + any resource, + any connected server + + + + github__create_issue + prefixed MCP tool, + pass-through + + + + verification gate + sha256 + size per file + frontmatter == entry + unlisted file → refuse + mismatch → skills/get, retry once + + + + load_skill("git-helper") + + load_skill("deploy" | URI) + + read_resource(server, uri) + + github__create_issue(…) + + + + read SKILL.md + from disk + + + + + resources/read, + origin-bound + + + + "dynamic" entry: + no manifest — + skip gate; files from + directory/read walk + or resources/list; + reads unverified + + + + resources/read — + raw content, no verify, + no skill activation + + + + tools/call + + + + startup: + walk skill dirs + → prompt list + + + + on connect: + skills/list + → cache → + prompt list + + + + Local filesystem + ~/.goose/skills · project · plugins + git-helper/SKILL.md + + references/… + + + + MCP server — declares io.modelcontextprotocol/skills (directoryRead?) + skills/list · skills/get · resources/read · resources/directory/read · tools/call + skill://deploy/SKILL.md + supporting files, served as resources + entry: {uri, frontmatter, resources: [{uri, digest, size}] | "dynamic"} + + + Both load_skill origins return the same framing — “# Loaded Skill: deploy (mcp skill from github)” — so origin stays visible to the model. + solid — model-initiated call  ·  dashed — discovery/metadata  ·  blue — what SEP-2640 adds over plain resources + diff --git a/experiments/sep-2640-github-mcp-server-surfaces.svg b/experiments/sep-2640-github-mcp-server-surfaces.svg new file mode 100644 index 0000000..9ff3456 --- /dev/null +++ b/experiments/sep-2640-github-mcp-server-surfaces.svg @@ -0,0 +1,96 @@ + + + + + + + + + + + + Agent host (Claude Code, Copilot, gemini-cli, …) + builds one skill registry, loads SKILL.md into model context + + + + regular skills: + scan checkout, + read files + from disk + + + + MCP (initialize, JSON-RPC) + + + + github-mcp-server — feature/agent-skills-v2 + declares capabilities.extensions[“io.modelcontextprotocol/skills”] = { directoryRead: true } + + + + + Tools + tools/list · tools/call + GitHub tools + issues, PRs, actions, … + + list_repo_skills + finds SKILL.md dirs in any + repo, returns skill:// URIs + the model hands to + resources/read or skills/get + + + + + Resources + resources/read · templates + repo://… content + branches, commits, tags, PRs + + skill://github/<name>/… + bundled skill files (embedded) + + skill://{owner}/{repo}/ + {skill}/{file} + repo-hosted skill files + + + + + Skills extension (SEP-2640) + skills/list + bundled catalog: frontmatter + + {uri, digest, size} per file + + skills/get + entry by URI — bundled or + repo-hosted, listed or not + + resources/directory/read + ls for any skill directory, + both namespaces + + + + skills/ directory in this repo + 28 regular Agent Skills — plain SKILL.md directories, + readable by any agent that scans the checkout; + the same files are embedded into the binary and + served over MCP as skill://github/<name>/… + + + Any GitHub repository + skills/*/SKILL.md · skills/{ns}/*/SKILL.md · + plugins/*/skills/*/SKILL.md · */SKILL.md + never in skills/list (unenumerable catalog) — + reached by URI via skills/get and resources/read + + + + go:embed (build time) + + GitHub API (on demand) + From 8c080b69bc39ef785afbc9846326413e68ee5d9b Mon Sep 17 00:00:00 2001 From: olaservo Date: Tue, 25 Aug 2026 08:35:35 -0700 Subject: [PATCH 3/6] chore: keep check-diagram skill local for now Claude-Session: https://claude.ai/code/session_017L3xguMgzVZQNrzu2AvKX2 Signed-off-by: olaservo --- .claude/skills/check-diagram/SKILL.md | 64 --------------------------- .gitignore | 1 + 2 files changed, 1 insertion(+), 64 deletions(-) delete mode 100644 .claude/skills/check-diagram/SKILL.md diff --git a/.claude/skills/check-diagram/SKILL.md b/.claude/skills/check-diagram/SKILL.md deleted file mode 100644 index ab71bf8..0000000 --- a/.claude/skills/check-diagram/SKILL.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -name: check-diagram -description: Check that a skills-over-MCP surfaces diagram (SVG) matches the latest commit of the implementation branch it describes. Works for any client or server implementation. Use when asked whether a diagram is accurate, current, or needs updating after a branch moved. ---- - -# Check a diagram against the branch tip - -Input: a path to an SVG in this repo (`experiments/**/*.svg`). Optional: `owner/repo@branch` if the diagram does not name one. - -## 1. Identify the implementation - -Nothing about the implementation is fixed in this skill. Find it from the material itself: - -- the SVG text usually names the branch (grep it for `feature/`, `-port`, `branch`); -- the README that links the diagram (`grep -rl --include=*.md`) names the repo as a GitHub URL and the branch; -- if the diagram has no README link yet, the nearest README above it in `experiments/` names the reference server and host. - -If neither the diagram nor a README names a repo and branch, ask; do not guess from local clones. - -## 2. Get the branch tip, not a local checkout - -Local clones and worktrees go stale. Always resolve the remote tip first: - -``` -git ls-remote https://github.com// refs/heads/ -``` - -Then read code at that exact commit. Either fetch into an existing clone (`git fetch origin ` then `git show origin/:` and `git grep origin/ -- `), or shallow-clone into the scratchpad (`git clone --depth 1 --branch `). Never read from a working tree without confirming `git rev-parse HEAD` equals the `ls-remote` sha. - -Record the sha and its date in the report. - -## 3. Extract the diagram's claims - -Pull the text out of the SVG: - -``` -grep -o ']*>[^<]*' | sed 's/]*>//' -``` - -Turn every concrete statement into a checkable claim. Typical claim types: - -- protocol methods and capability declarations (`skills/list`, `capabilities.extensions["io.modelcontextprotocol/skills"] = { directoryRead: true }`) -- wire shapes and field names (`{uri, digest, size}`, `"dynamic"`, `resultType`) -- URI patterns (`skill://github//…`, `skill://{owner}/{repo}/{skill}/{file}`) -- tool names and what they route to (`load_skill`, `read_resource`, `list_repo_skills`) -- gates and behaviours (digest + size check, frontmatter identity, unlisted file refused, one `skills/get` retry, per-server `skills_enabled`) -- counts (`28 regular Agent Skills`) and the branch name itself - -Layout, colour, and wording that carries no fact are not claims. - -## 4. Verify each claim in the code at the tip - -For each claim find the code that implements it and cite `path:line` at the checked sha. Search, do not recall: method names in request handlers or dispatch, field names in the wire structs, URI prefixes as string constants, counts by listing the directory. A claim is one of: - -- **matches** — code at the tip does what the diagram says; cite the evidence -- **stale** — the tip does something different; say what, cite the evidence -- **missing** — the tip has a surface the diagram does not show and a reader would expect on it -- **unverifiable** — could not find evidence either way; say what was searched - -Then compare against the SEP baseline the branch claims to track (`docs/sep-draft-skills-extension.md` header pins the commit; the PR is authoritative per AGENTS.md). Flag any diagram claim that matches the code but no longer matches the spec, since the code will change next. - -## 5. Report - -Plain markdown, one table row per claim, then the branch sha/date and the SEP commit compared. Below the table, list the exact SVG text edits needed for each stale or missing item. Do not edit the SVG unless asked; the diagrams are hand-authored and the user updates them. diff --git a/.gitignore b/.gitignore index d6b130c..751cc17 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ .claude/settings.local.json +.claude/skills/check-diagram/ From 9a3bac858e15ac1a3fe2f414fa9ff31c2745e30a Mon Sep 17 00:00:00 2001 From: olaservo Date: Tue, 25 Aug 2026 08:37:06 -0700 Subject: [PATCH 4/6] chore: drop skill ignore entry Claude-Session: https://claude.ai/code/session_017L3xguMgzVZQNrzu2AvKX2 Signed-off-by: olaservo --- .gitignore | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitignore b/.gitignore index 751cc17..d6b130c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1 @@ .claude/settings.local.json -.claude/skills/check-diagram/ From b1e3d39d3fa606e3541893093ff0edbc0f7cc5f0 Mon Sep 17 00:00:00 2001 From: olaservo Date: Tue, 25 Aug 2026 20:53:55 -0700 Subject: [PATCH 5/6] feat(experiments): goose scenarios against the skills-over-mcp-demo Space MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three scenarios against https://olaservo-skills-over-mcp-demo.hf.space/mcp: discovery + load, the unlisted skill reached through the server's instructions pointer and skills/get, and a supporting file loaded through the digest + size gate. All pass with goose fdf9ae6c. Harness: bearer_cmd is now opt-in (absent means no Authorization header), a resources_read expectation, tool lines with an owning extension ("▸ read_resource extensionmanager") are parsed, and `loaded` accepts the "# Loaded:" supporting-file frame. Claude-Session: https://claude.ai/code/session_017L3xguMgzVZQNrzu2AvKX2 Signed-off-by: olaservo --- experiments/README.md | 2 + experiments/goose/README.md | 11 +- ...mo-discovery-and-load.20260825-205208.json | 132 ++++++++++++++++++ .../demo-supporting-file.20260825-205332.json | 102 ++++++++++++++ .../demo-unlisted-skill.20260825-205215.json | 82 +++++++++++ experiments/goose/run_scenario.py | 45 +++--- .../scenarios/demo-discovery-and-load.yaml | 13 ++ .../goose/scenarios/demo-supporting-file.yaml | 11 ++ .../goose/scenarios/demo-unlisted-skill.yaml | 11 ++ .../goose/scenarios/discovery-and-load.yaml | 1 + experiments/goose/scenarios/uri-load.yaml | 1 + 11 files changed, 388 insertions(+), 23 deletions(-) create mode 100644 experiments/goose/results/demo-discovery-and-load.20260825-205208.json create mode 100644 experiments/goose/results/demo-supporting-file.20260825-205332.json create mode 100644 experiments/goose/results/demo-unlisted-skill.20260825-205215.json create mode 100644 experiments/goose/scenarios/demo-discovery-and-load.yaml create mode 100644 experiments/goose/scenarios/demo-supporting-file.yaml create mode 100644 experiments/goose/scenarios/demo-unlisted-skill.yaml diff --git a/experiments/README.md b/experiments/README.md index 36cba24..13eb1f4 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -12,4 +12,6 @@ Reference server for all harnesses: [github/github-mcp-server#3046](https://gith The 28 bundled skills are plain `SKILL.md` directories in the checkout, embedded at build time and served as `skill://github//…`; `skills/list` enumerates only these. Skills in any GitHub repository are reachable by URI (`skill://{owner}/{repo}/{skill}/{file}`) through `skills/get`, `resources/read`, and `resources/directory/read`, with `list_repo_skills` as the tool that finds them, but they never appear in `skills/list`. +Second server: [skills-over-mcp-demo](https://github.com/olaservo/skills-over-mcp-demo), a small SEP-2640 server on the v2 TypeScript SDK, live and unauthenticated at `https://olaservo-skills-over-mcp-demo.hf.space/mcp`. It adds what the GitHub server lacks: an unlisted skill reachable only through the server's `instructions` pointer and `skills/get`, supporting files (so the digest and size gate is exercised), and a multi-segment skill path. Free Spaces sleep when idle; the harness's wire probe wakes it before goose connects. + Conventions shared by all harnesses: the runner never spawns the MCP server — it connects to an already-running server at the scenario's `mcp_server.endpoint` — and the scenario file is always an explicit `--scenario ` argument. diff --git a/experiments/goose/README.md b/experiments/goose/README.md index 865b54b..5a4a418 100644 --- a/experiments/goose/README.md +++ b/experiments/goose/README.md @@ -12,16 +12,18 @@ How goose (`skills-sep-2640-port`) exposes SEP-2640 to the model: local and MCP- - A goose build with skills-over-MCP support: branch `skills-sep-2640-port` of https://github.com/olaservo/goose (`cargo build -p goose-cli`). Point the harness at the binary with `--goose` or `GOOSE_BIN`. Use `fdf9ae6c` or later: the 2026-08-21/24 upstream merges left `OAuthStepUpClient` without `skills_list`/`skills_get`/`directory_read` forwarding, so every streamable HTTP server reported `Transport closed`. - A running SEP-2640 server. Reference: draft PR [github/github-mcp-server#3046](https://github.com/github/github-mcp-server/pull/3046), branch `olaservo:feature/agent-skills-v2` (`09ea2f9f` or later for the b405ba5 `size`/`"dynamic"` contract) — `go build -o github-mcp-server-skills.exe ./cmd/github-mcp-server`, then `./github-mcp-server-skills.exe http --port 8082`. The harness never starts the server; it connects to the scenario's `mcp_server.endpoint`. -- Auth: the scenario's `mcp_server.bearer_cmd` (default `gh auth token`) is run once and sent as a `Authorization: Bearer …` header by both the wire probe and goose. +- Auth: when the scenario sets `mcp_server.bearer_cmd`, it is run once and its output sent as a `Authorization: Bearer …` header by both the wire probe and goose. Omit it for unauthenticated servers. - An LLM provider goose can use. The harness writes an isolated goose config (nothing in your real config is touched); provider credentials come from the system keyring or provider env vars as usual. Defaults: `anthropic` / `claude-sonnet-4-6`, overridable per scenario. +Scenarios named `demo-*` run against the public demo Space instead (see [`../README.md`](../README.md)); they need no server or auth setup. Observed 2026-08-25: the live Space negotiates `2025-11-25` and its `skills/list` carries no `ttlMs`/`cacheScope`, unlike the demo repo's README claims for its current commit; the deployment may be older than the repo. + ## Run ``` uv run --with pyyaml run_scenario.py --scenario scenarios/discovery-and-load.yaml ``` -The result JSON lands in `results/`, the isolated goose config root in a temp dir (path printed; it contains the bearer token, delete when done). Exit code is 0 only if every check passed. +The result JSON lands in `results/`, the isolated goose config root in a temp dir (path printed; it contains the bearer token when one was used, delete when done). Exit code is 0 only if every check passed. ## Scenario format @@ -29,7 +31,7 @@ The result JSON lands in `results/`, the isolated goose config root in a temp di name: discovery-and-load mcp_server: endpoint: http://localhost:8082/mcp - bearer_cmd: gh auth token # optional, this is the default + bearer_cmd: gh auth token # optional; omit for unauthenticated servers provider: anthropic # optional model: claude-sonnet-4-6 # optional prompt: | @@ -38,8 +40,9 @@ expect: extension_declared: true # server declares io.modelcontextprotocol/skills resources_sized: true # every manifest element in skills/list carries an integer size discovered: [create-issue] # ⊆ wire skills/list names AND each appears in the reply text - loaded: [create-issue] # ⊆ load_skill calls whose result was framed skill content (`# Loaded Skill:`), not an error + loaded: [create-issue] # ⊆ load_skill calls whose result was framed content (`# Loaded Skill:` / `# Loaded:`), not an error tools: [load_skill] # optional, ⊆ tool names goose called + resources_read: [skill://…] # optional, ⊆ the `uri` args of read_resource calls ``` Every `expect` key is optional; omitted keys are recorded but not asserted. The harness grades nothing else — it logs the full tool-call trace and the reply tail so a human can read what actually happened. diff --git a/experiments/goose/results/demo-discovery-and-load.20260825-205208.json b/experiments/goose/results/demo-discovery-and-load.20260825-205208.json new file mode 100644 index 0000000..5d83915 --- /dev/null +++ b/experiments/goose/results/demo-discovery-and-load.20260825-205208.json @@ -0,0 +1,132 @@ +{ + "scenario": "demo-discovery-and-load", + "timestamp": "2026-08-25T20:52:08", + "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "tabletop-dice", + "uri": "skill://dice-roller/tabletop-dice/SKILL.md", + "resources": 2, + "sized": true + }, + { + "name": "mcp-glossary", + "uri": "skill://mcp-glossary/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "release-notes-writer", + "uri": "skill://release-notes-writer/SKILL.md", + "resources": 2, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "tabletop-dice" + } + } + ], + "load_skill_calls": [ + "tabletop-dice" + ], + "skills_loaded": [ + "tabletop-dice" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-pkf8edyi\\wd\n L L goose is ready\nHere are the skills listed as coming from connected MCP servers in my system instructions:\n\n- mcp-glossary (scenario_server)\n- release-notes-writer (scenario_server)\n- tabletop-dice (scenario_server)\n\n---\n\nNow loading the skill:\n ────────────────────────────────────────\n ▸ load_skill\n name: tabletop-dice\n\nThe first markdown heading line of the loaded content is:\n\n**`# Tabletop Dice`**\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "resources_sized", + "pass": true, + "detail": { + "manifest_entries": 3, + "unsized": [] + } + }, + { + "check": "discovered on wire: tabletop-dice", + "pass": true, + "detail": [ + "tabletop-dice", + "mcp-glossary", + "release-notes-writer" + ] + }, + { + "check": "enumerated by model: tabletop-dice", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: mcp-glossary", + "pass": true, + "detail": [ + "tabletop-dice", + "mcp-glossary", + "release-notes-writer" + ] + }, + { + "check": "enumerated by model: mcp-glossary", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "discovered on wire: release-notes-writer", + "pass": true, + "detail": [ + "tabletop-dice", + "mcp-glossary", + "release-notes-writer" + ] + }, + { + "check": "enumerated by model: release-notes-writer", + "pass": true, + "detail": "searched reply text" + }, + { + "check": "loaded: tabletop-dice", + "pass": true, + "detail": { + "loaded": [ + "tabletop-dice" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/results/demo-supporting-file.20260825-205332.json b/experiments/goose/results/demo-supporting-file.20260825-205332.json new file mode 100644 index 0000000..e93e8ab --- /dev/null +++ b/experiments/goose/results/demo-supporting-file.20260825-205332.json @@ -0,0 +1,102 @@ +{ + "scenario": "demo-supporting-file", + "timestamp": "2026-08-25T20:53:32", + "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "tabletop-dice", + "uri": "skill://dice-roller/tabletop-dice/SKILL.md", + "resources": 2, + "sized": true + }, + { + "name": "mcp-glossary", + "uri": "skill://mcp-glossary/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "release-notes-writer", + "uri": "skill://release-notes-writer/SKILL.md", + "resources": 2, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "tabletop-dice" + } + }, + { + "tool": "load_skill", + "args": { + "name": "tabletop-dice/references/dice-notation.md" + } + } + ], + "load_skill_calls": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "skills_loaded": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-eoa6k3on\\wd\n L L goose is ready\nI'll load the tabletop-dice skill first, then follow its instructions for loading the supporting file.\n ────────────────────────────────────────\n ▸ load_skill\n name: tabletop-dice\n\nNow I'll load the supporting file as instructed:\n ────────────────────────────────────────\n ▸ load_skill\n name: tabletop-dice/references/dice-notation.md\n\nThe first markdown heading line of the supporting file is:\n\n```\n# Dice Notation Reference\n```\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "loaded: tabletop-dice", + "pass": true, + "detail": { + "loaded": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "failed": [] + } + }, + { + "check": "loaded: tabletop-dice/references/dice-notation.md", + "pass": true, + "detail": { + "loaded": [ + "tabletop-dice", + "tabletop-dice/references/dice-notation.md" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/results/demo-unlisted-skill.20260825-205215.json b/experiments/goose/results/demo-unlisted-skill.20260825-205215.json new file mode 100644 index 0000000..14620f3 --- /dev/null +++ b/experiments/goose/results/demo-unlisted-skill.20260825-205215.json @@ -0,0 +1,82 @@ +{ + "scenario": "demo-unlisted-skill", + "timestamp": "2026-08-25T20:52:15", + "endpoint": "https://olaservo-skills-over-mcp-demo.hf.space/mcp", + "server": { + "extension_declared": true, + "extension_settings": { + "directoryRead": true + }, + "protocol_version": "2025-11-25", + "result_type": null, + "skills": [ + { + "name": "tabletop-dice", + "uri": "skill://dice-roller/tabletop-dice/SKILL.md", + "resources": 2, + "sized": true + }, + { + "name": "mcp-glossary", + "uri": "skill://mcp-glossary/SKILL.md", + "resources": 1, + "sized": true + }, + { + "name": "release-notes-writer", + "uri": "skill://release-notes-writer/SKILL.md", + "resources": 2, + "sized": true + } + ] + }, + "run": { + "tools_used": [ + "load_skill" + ], + "tool_calls": [ + { + "tool": "load_skill", + "args": { + "name": "skill://secret-menu/SKILL.md" + } + } + ], + "load_skill_calls": [ + "skill://secret-menu/SKILL.md" + ], + "skills_loaded": [ + "skill://secret-menu/SKILL.md" + ], + "load_failures": [], + "resources_read": [], + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260826_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-4l0b61r8\\wd\n L L goose is ready\n\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://secret-menu/SKILL.md\n\nThe first markdown heading line of the loaded content is:\n\n```\n# Secret Menu\n```\n" + }, + "checks": [ + { + "check": "extension_declared", + "pass": true, + "detail": { + "directoryRead": true + } + }, + { + "check": "loaded: skill://secret-menu/SKILL.md", + "pass": true, + "detail": { + "loaded": [ + "skill://secret-menu/SKILL.md" + ], + "failed": [] + } + }, + { + "check": "tool used: load_skill", + "pass": true, + "detail": [ + "load_skill" + ] + } + ], + "passed": true +} \ No newline at end of file diff --git a/experiments/goose/run_scenario.py b/experiments/goose/run_scenario.py index 9552b20..a9b8a7b 100644 --- a/experiments/goose/run_scenario.py +++ b/experiments/goose/run_scenario.py @@ -27,15 +27,13 @@ def rpc(endpoint, token, method, params, req_id): body = json.dumps( {"jsonrpc": "2.0", "id": req_id, "method": method, "params": params} ).encode() - req = urllib.request.Request( - endpoint, - data=body, - headers={ - "Content-Type": "application/json", - "Accept": "application/json, text/event-stream", - "Authorization": f"Bearer {token}", - }, - ) + headers = { + "Content-Type": "application/json", + "Accept": "application/json, text/event-stream", + } + if token: + headers["Authorization"] = f"Bearer {token}" + req = urllib.request.Request(endpoint, data=body, headers=headers) raw = urllib.request.urlopen(req, timeout=30).read().decode("utf-8", "replace") for line in raw.splitlines(): if line.startswith("data: "): @@ -109,17 +107,21 @@ def write_goose_config(root, scenario, token): "uri": endpoint, "description": "scenario MCP server", "timeout": 300, - "headers": {"Authorization": f"Bearer {token}"}, }, }, } + if token: + config["extensions"]["scenario_server"]["headers"] = { + "Authorization": f"Bearer {token}" + } config_dir = os.path.join(root, "config") os.makedirs(config_dir) with open(os.path.join(config_dir, "config.yaml"), "w", encoding="utf-8") as f: yaml.safe_dump(config, f) -TOOL_LINE = re.compile(r"^\s*[▸>]\s+(\S+)\s*$") +# "▸ load_skill" or "▸ read_resource extensionmanager" (tool, then owning extension) +TOOL_LINE = re.compile(r"^\s*[▸>]\s+(\S+)(?:\s+\S+)?\s*$") ARG_LINE = re.compile(r"^\s{2,}(\w+): (.*)$") @@ -196,10 +198,12 @@ def main(): scenario = yaml.safe_load(f) expect = scenario.get("expect", {}) or {} - bearer_cmd = scenario["mcp_server"].get("bearer_cmd", "gh auth token") - token = subprocess.run( - bearer_cmd, shell=True, capture_output=True, text=True, check=True - ).stdout.strip() + bearer_cmd = scenario["mcp_server"].get("bearer_cmd") + token = "" + if bearer_cmd: + token = subprocess.run( + bearer_cmd, shell=True, capture_output=True, text=True, check=True + ).stdout.strip() server = probe_server(scenario["mcp_server"]["endpoint"], token) wire_names = [s["name"] for s in server["skills"]] @@ -208,7 +212,7 @@ def main(): workdir = os.path.join(root, "wd") os.makedirs(workdir) write_goose_config(root, scenario, token) - print(f"goose config root (contains bearer token): {root}") + print(f"goose config root{' (contains bearer token)' if token else ''}: {root}") output = run_goose(args.goose, scenario, root, workdir) @@ -218,13 +222,14 @@ def main(): load_results = [ text for tid, (name, text) in tool_results(root).items() if name == "load_skill" ] - # A load only counts when goose returned framed skill content, not an error. + # A load only counts when goose returned framed content, not an error: + # "# Loaded Skill:" for a SKILL.md, "# Loaded:" for a supporting file. skills_loaded = [ - n for n, text in zip(load_calls, load_results) if text.startswith("# Loaded Skill:") + n for n, text in zip(load_calls, load_results) if text.startswith("# Loaded") ] load_failures = [ (n, text[:200]) for n, text in zip(load_calls, load_results) - if not text.startswith("# Loaded Skill:") + if not text.startswith("# Loaded") ] resources_read = [a["uri"] for n, a in calls if n == "read_resource" and "uri" in a] @@ -250,6 +255,8 @@ def check(name, passed, detail): {"loaded": skills_loaded, "failed": load_failures}) for name in expect.get("tools", []): check(f"tool used: {name}", name in tools_used, tools_used) + for uri in expect.get("resources_read", []): + check(f"resource read: {uri}", uri in resources_read, resources_read) result = { "scenario": scenario["name"], diff --git a/experiments/goose/scenarios/demo-discovery-and-load.yaml b/experiments/goose/scenarios/demo-discovery-and-load.yaml new file mode 100644 index 0000000..f40f969 --- /dev/null +++ b/experiments/goose/scenarios/demo-discovery-and-load.yaml @@ -0,0 +1,13 @@ +name: demo-discovery-and-load +mcp_server: + endpoint: https://olaservo-skills-over-mcp-demo.hf.space/mcp +prompt: | + First: list every skill name that your system instructions say comes from a connected MCP server, one per line, exactly as written. + Second: call the load_skill tool with name 'tabletop-dice' and reply with the first markdown heading line of the loaded content. + Use no tools other than load_skill. +expect: + extension_declared: true + resources_sized: true + discovered: [tabletop-dice, mcp-glossary, release-notes-writer] + loaded: [tabletop-dice] + tools: [load_skill] diff --git a/experiments/goose/scenarios/demo-supporting-file.yaml b/experiments/goose/scenarios/demo-supporting-file.yaml new file mode 100644 index 0000000..d1886d4 --- /dev/null +++ b/experiments/goose/scenarios/demo-supporting-file.yaml @@ -0,0 +1,11 @@ +name: demo-supporting-file +mcp_server: + endpoint: https://olaservo-skills-over-mcp-demo.hf.space/mcp +prompt: | + Call the load_skill tool with name 'tabletop-dice'. The loaded content ends with a Supporting Files section that shows exactly how to load its supporting file; load it that way. + Reply with the first markdown heading line of the supporting file. + Use no tools other than load_skill. +expect: + extension_declared: true + loaded: [tabletop-dice, tabletop-dice/references/dice-notation.md] + tools: [load_skill] diff --git a/experiments/goose/scenarios/demo-unlisted-skill.yaml b/experiments/goose/scenarios/demo-unlisted-skill.yaml new file mode 100644 index 0000000..591e680 --- /dev/null +++ b/experiments/goose/scenarios/demo-unlisted-skill.yaml @@ -0,0 +1,11 @@ +name: demo-unlisted-skill +mcp_server: + endpoint: https://olaservo-skills-over-mcp-demo.hf.space/mcp +prompt: | + A connected MCP server's instructions point to a skill for coffee recommendations that is not in your skill list. + Call the load_skill tool with that skill's URI exactly as the instructions give it, and reply with the first markdown heading line of the loaded content. + Use no tools other than load_skill. +expect: + extension_declared: true + loaded: ["skill://secret-menu/SKILL.md"] + tools: [load_skill] diff --git a/experiments/goose/scenarios/discovery-and-load.yaml b/experiments/goose/scenarios/discovery-and-load.yaml index 18840dd..6425edb 100644 --- a/experiments/goose/scenarios/discovery-and-load.yaml +++ b/experiments/goose/scenarios/discovery-and-load.yaml @@ -1,6 +1,7 @@ name: discovery-and-load mcp_server: endpoint: http://localhost:8082/mcp + bearer_cmd: gh auth token prompt: | First: list every skill name that your system instructions say comes from a connected MCP server, one per line, exactly as written. Second: call the load_skill tool with name 'create-issue' and reply with the first markdown heading line of the loaded content. diff --git a/experiments/goose/scenarios/uri-load.yaml b/experiments/goose/scenarios/uri-load.yaml index 136e306..4e4792d 100644 --- a/experiments/goose/scenarios/uri-load.yaml +++ b/experiments/goose/scenarios/uri-load.yaml @@ -1,6 +1,7 @@ name: uri-load mcp_server: endpoint: http://localhost:8082/mcp + bearer_cmd: gh auth token prompt: | Call the load_skill tool with name 'skill://github/create-pr/SKILL.md' (the full URI, exactly as given). Reply with the first markdown heading line of the loaded content and the origin server named in the loaded header. From b20fdbdf687d0f3ef90f7f4cd48bd7b44b29ee38 Mon Sep 17 00:00:00 2001 From: olaservo Date: Thu, 27 Aug 2026 19:02:21 -0700 Subject: [PATCH 6/6] chore(experiments): regenerate GitHub server results with the final harness Same goose (fdf9ae6c) and server (7d4ae593); both scenarios pass. Claude-Session: https://claude.ai/code/session_017L3xguMgzVZQNrzu2AvKX2 Signed-off-by: olaservo --- ...24-212736.json => discovery-and-load.20260827-190206.json} | 4 ++-- ...oad.20260824-212742.json => uri-load.20260827-190213.json} | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) rename experiments/goose/results/{discovery-and-load.20260824-212736.json => discovery-and-load.20260827-190206.json} (95%) rename experiments/goose/results/{uri-load.20260824-212742.json => uri-load.20260827-190213.json} (90%) diff --git a/experiments/goose/results/discovery-and-load.20260824-212736.json b/experiments/goose/results/discovery-and-load.20260827-190206.json similarity index 95% rename from experiments/goose/results/discovery-and-load.20260824-212736.json rename to experiments/goose/results/discovery-and-load.20260827-190206.json index 2389b16..2bfb6a9 100644 --- a/experiments/goose/results/discovery-and-load.20260824-212736.json +++ b/experiments/goose/results/discovery-and-load.20260827-190206.json @@ -1,6 +1,6 @@ { "scenario": "discovery-and-load", - "timestamp": "2026-08-24T21:27:36", + "timestamp": "2026-08-27T19:02:06", "endpoint": "http://localhost:8082/mcp", "server": { "extension_declared": true, @@ -200,7 +200,7 @@ ], "load_failures": [], "resources_read": [], - "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260825_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-7g8x1h6y\\wd\n L L goose is ready\nHere are the skills from connected MCP servers, exactly as written in my system instructions:\n\n- address-pr-feedback (scenario_server)\n- browse-discussions (scenario_server)\n- contribute-oss (scenario_server)\n- create-issue (scenario_server)\n- create-pr (scenario_server)\n- debug-ci (scenario_server)\n- delegate-to-copilot (scenario_server)\n- discover-github (scenario_server)\n- discover-mcp-skills (scenario_server)\n- explore-repo (scenario_server)\n- fix-dependabot (scenario_server)\n- get-context (scenario_server)\n- handle-notifications (scenario_server)\n- manage-labels (scenario_server)\n- manage-project (scenario_server)\n- manage-repo (scenario_server)\n- manage-sub-issues (scenario_server)\n- merge-pr (scenario_server)\n- prepare-release (scenario_server)\n- research-vulnerability (scenario_server)\n- review-pr (scenario_server)\n- search-code (scenario_server)\n- security-audit (scenario_server)\n- self-review-pr (scenario_server)\n- share-snippet (scenario_server)\n- trace-history (scenario_server)\n- triage-issues (scenario_server)\n- trigger-workflow (scenario_server)\n\nNow loading the `create-issue` skill:\n ────────────────────────────────────────\n ▸ load_skill\n name: create-issue\n\nThe first markdown heading line of the loaded content is:\n\n**`# Create Issue`**\n" + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260828_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-quott2po\\wd\n L L goose is ready\nHere are the skills from connected MCP servers, exactly as written in my system instructions:\n\n- address-pr-feedback (scenario_server)\n- browse-discussions (scenario_server)\n- contribute-oss (scenario_server)\n- create-issue (scenario_server)\n- create-pr (scenario_server)\n- debug-ci (scenario_server)\n- delegate-to-copilot (scenario_server)\n- discover-github (scenario_server)\n- discover-mcp-skills (scenario_server)\n- explore-repo (scenario_server)\n- fix-dependabot (scenario_server)\n- get-context (scenario_server)\n- handle-notifications (scenario_server)\n- manage-labels (scenario_server)\n- manage-project (scenario_server)\n- manage-repo (scenario_server)\n- manage-sub-issues (scenario_server)\n- merge-pr (scenario_server)\n- prepare-release (scenario_server)\n- research-vulnerability (scenario_server)\n- review-pr (scenario_server)\n- search-code (scenario_server)\n- security-audit (scenario_server)\n- self-review-pr (scenario_server)\n- share-snippet (scenario_server)\n- trace-history (scenario_server)\n- triage-issues (scenario_server)\n- trigger-workflow (scenario_server)\n\n---\n\nNow loading the `create-issue` skill:\n ────────────────────────────────────────\n ▸ load_skill\n name: create-issue\n\nThe first markdown heading line of the loaded content is:\n\n**`# Create Issue`**\n" }, "checks": [ { diff --git a/experiments/goose/results/uri-load.20260824-212742.json b/experiments/goose/results/uri-load.20260827-190213.json similarity index 90% rename from experiments/goose/results/uri-load.20260824-212742.json rename to experiments/goose/results/uri-load.20260827-190213.json index 64527f9..09f0f29 100644 --- a/experiments/goose/results/uri-load.20260824-212742.json +++ b/experiments/goose/results/uri-load.20260827-190213.json @@ -1,6 +1,6 @@ { "scenario": "uri-load", - "timestamp": "2026-08-24T21:27:42", + "timestamp": "2026-08-27T19:02:13", "endpoint": "http://localhost:8082/mcp", "server": { "extension_declared": true, @@ -200,7 +200,7 @@ ], "load_failures": [], "resources_read": [], - "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260825_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-5bfr0zjv\\wd\n L L goose is ready\nI'll load the skill using the exact URI provided.\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://github/create-pr/SKILL.md\n\nHere are the two requested pieces of information:\n\n- **First markdown heading line:** `# Create Pull Request`\n- **Origin server named in the loaded header:** `scenario_server`\n" + "output_tail": "\n __( O)> ● new session · anthropic claude-sonnet-4-6\n \\____) 20260828_1 · C:\\Users\\johnn\\AppData\\Local\\Temp\\goose-scenario-jip200wl\\wd\n L L goose is ready\n\n ────────────────────────────────────────\n ▸ load_skill\n name: skill://github/create-pr/SKILL.md\n\nHere are the two pieces of information you asked for:\n\n- **First markdown heading line:** `# Create Pull Request`\n- **Origin server named in the loaded header:** `scenario_server`\n" }, "checks": [ {