diff --git a/AGENTS.md b/AGENTS.md index 1df05e1..b3d6a8a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,9 +5,17 @@ Pi-Package (`pi-package`), das Firstmate bei Crew-Routing und Quota-Balance unte ## Was dieses Repo ist - Installierbare Extension: `package.json` → `"pi": { "extensions": ["./index.ts"] }` -- Tools: `crew_route`, `crew_balance`, `crew_apply_dispatch`, `crew_suggest_primary` -- Knowledge: `knowledge/*.json` (Task-Klassen, Provider, Profile) -- Logik in `src/` +- Tools: `crew_route`, `crew_balance`, `crew_apply_dispatch`, `crew_suggest_primary`, `crew_discover`, `crew_evidence`, `crew_update_check` +- Knowledge: Drei-Ebenen System + - Basis: `knowledge/*.json` (Task-Klassen, Provider, Profile) + - Hersteller: `knowledge/manufacturers/` (Official Docs, Manufacturer Claims) + - Benchmarks: `knowledge/benchmarks/` (Artificial Analysis, HumanEval, etc.) + - Lokale Evidenz: `knowledge/local/` (no-mistakes Outcomes, privacy-conscious) +- Logik in `src/`: + - `discovery.ts`: Live Model Discovery (Pi, Claude, Codex, Grok, Kimi) + - `evidence.ts`: Lokale Evidenz-Sammlung aus no-mistakes + - `knowledge-layers.ts`: Drei-Ebenen Knowledge System mit Conflict Resolution + - `update.ts`: Compatibility Checks und Firstmate-Version-Awareness ## Was es nicht ist @@ -23,12 +31,25 @@ Autoritative Firstmate-Doku liegt im Firstmate-Home (nicht hier kopieren): - Spawn-Flags → `bin/fm-spawn.sh --harness/--model/--effort` - Quota-Daten → `quota-axi --json` (Skill `quota-axi`) +## Architektur-Prinzipien + +- **Firstmate bleibt Authority**: crew-knowledge liefert nur Empfehlungen +- **Kein Competing Dispatch Engine**: Nutzt Firstmate's `quota-array-dispatch` +- **Respektiert config/crew-dispatch.json**: Natural Language Rules bleiben authoritative +- **Provider ≠ Model ≠ Harness**: Klare Trennung, kein Inferieren von Provider aus Model-Namen +- **Herdr ist Infrastructure**: Backend-Wahl (tmux, Herdr, etc.) beeinflusst nicht Model-Qualität +- **Discovery Cache**: 15min TTL, force-refresh möglich +- **Privacy-First Evidence**: Nur Metriken, KEINE Code-Inhalte, KEINE Prompts + ## Lokale Konventionen - User-facing Docs: Deutsch; kurze English-Section in README ok - Commits/PR: Deutsch, ohne AI-Co-Author-Meta - Writes nur unter dem aufgelösten Home/config (siehe README) und nur bei explizitem Tool-Call (`dryRun=false`) - Tests: `npm test` (Node built-in test runner, Fixtures, kein Netz) +- Evidence Collection: Automatisch aus no-mistakes, manuell via `recordEvidence()` oder `recordFromNoMistakesOutcome()` +- Discovery Cache: `~/.cache/firstmate-crew-knowledge/discovery.json` +- Local Evidence: `knowledge/local/evidence.json` und `knowledge/local/aggregated.json` ## Maintaining this file diff --git a/README.md b/README.md index b6b7945..dbfeea6 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,17 @@ # firstmate-crew-knowledge -Pi-Extension für [Firstmate](https://github.com/thelad-dev): empfiehlt pro Task-Klasse konkrete `{harness, model, effort}`-Profile und balanciert die Nutzung über die Captain-Subscriptions **Cursor**, **Claude**, **ChatGPT/Codex** und **Grok (xAI)**. +Pi-Extension für [Firstmate](https://github.com/thelad-dev): empfiehlt pro Task-Klasse konkrete `{harness, model, effort}`-Profile und balanciert die Nutzung über die Captain-Subscriptions **Cursor**, **Claude**, **ChatGPT/Codex**, **Grok (xAI)** und **Kimi**. > **Wichtig:** Dieses Paket **ersetzt weder Firstmate-Urteil noch `fm-spawn`**. Firstmate bleibt Owner von Intake, `quota-array-dispatch` und dem finalen Spawn. Die Tools liefern nur Empfehlungen, Spawn-Flags und optional eine `crew-dispatch.json`. +## Features + +- **Live Model Discovery**: Automatische Erkennung verfügbarer Modelle von Pi, Claude, Codex, Grok, Kimi mit Caching +- **Drei-Ebenen Knowledge System**: Hersteller-Fakten, externe Benchmarks, lokale Crew-Evidenz +- **Lokale Evidenz-Sammlung**: Automatisches Tracking von no-mistakes Outcomes (Privacy-conscious: keine Code-Inhalte, keine Prompts) +- **Selbstaktualisierung**: Kompatibilitätsprüfung mit Firstmate-Updates +- **Quota-Balance**: Live-Quota-Ranking über alle Subscriptions + ## Install ```bash @@ -31,6 +39,9 @@ Installable Pi package (`pi-package`) that classifies crew work, recommends `--h | Tool `crew_balance` | Live-Quota sortiert (Headroom/Runway), degraded wenn `quota-axi` fehlt | | Tool `crew_apply_dispatch` | Erzeugt Firstmate-`crew-dispatch.json` (Default: **dry-run**; `merge` hängt fehlende `when`-Regeln an und verweigert unlesbare/ungültige Bestandsdateien) | | Tool `crew_suggest_primary` | Listet/sucht authentifizierte Pi-Session-Modelle (`modelRegistry` / `scopedModels`) und empfiehlt ein Primary-Switch-Ziel nur darunter (kein Model-Switch, keine Crew-Panes) | +| Tool `crew_discover` | Live-Modell-Discovery von Pi, Claude, Codex, Grok, Kimi mit 15min Cache | +| Tool `crew_evidence` | Lokale Crew-Evidenz aus no-mistakes Outcomes anzeigen (aggregiert nach Task-Klasse/Modell) | +| Tool `crew_update_check` | Kompatibilitätsprüfung mit aktueller Firstmate-Version | | Command `/crew-route …` | Slash-Einstieg: vollständige `crew_route`-Ausgabe via Widget/Notify/Status (ohne Truncation, ohne Editor) | ### Empfohlener Ablauf im Primary @@ -61,10 +72,35 @@ Ohne aufgelöstes Home (`FM_HOME` oder Fallback `FM_ROOT_OVERRIDE`) nur Dry-Run/ Datengetrieben unter [`knowledge/`](knowledge/): +### Basis-Knowledge (Ebene 0) - `task-classes.json` — trivial_fix, standard_ship, hard_multi_file, research_web_live, scout_audit -- `providers.json` — Rollen/Stärken Claude · Codex · Cursor · Grok, Pi-Model-Muster (`provider/model`) +- `providers.json` — Rollen/Stärken Claude · Codex · Cursor · Grok · Kimi, Pi-Model-Muster (`provider/model`) - `profiles.json` — konkrete Profile + Dispatch-Template +### Drei-Ebenen Knowledge System + +#### A. Hersteller-Fakten (`knowledge/manufacturers/`) +- Offizielle Dokumentation von Anthropic, OpenAI, xAI, Moonshot AI +- Source: `official-docs` | `manufacturer-claim` +- Confidence: `high` (official-docs) | `low` (manufacturer-claim) + +#### B. Externe Benchmarks (`knowledge/benchmarks/`) +- Artificial Analysis, HumanEval, MBPP, Coding-Benchmarks +- Source: `external-benchmark` +- Confidence: `medium` + +#### C. Lokale Crew-Evidenz (`knowledge/local/`) +- Automatisch gesammelt aus no-mistakes Outcomes +- Source: `local` +- Confidence: `measured` +- **Privacy:** Nur Metriken (Task-Klasse, Harness, Provider, Modell, Effort, Duration, Success, Tests, CI, Rework) – keine Code-Inhalte, keine Prompts + +### Source-Konflikt-Hierarchie + +- **"Was ist verfügbar?"**: Live Discovery > Auth State > Manufacturer Website +- **"Was kann es?"**: Official Docs > Website > Discovery +- **"Wie gut?"**: Local Evidence > External Benchmarks > Manufacturer Claims + Tabellen aktualisieren, ohne Scorer-Code anzufassen. ## Fail-closed / Degraded @@ -81,6 +117,24 @@ npm test Tests laufen ohne Netzwerk gegen Fixture-JSON unter `tests/fixtures/`. +### Test Coverage + +- **Phase 1 (Discovery)**: `tests/discovery.test.ts` – Live Model Discovery, Caching, Staleness +- **Phase 2 (Knowledge Layers)**: `tests/knowledge-layers.test.ts` – Drei-Ebenen System, Conflict Resolution +- **Phase 3 (Update)**: `tests/update.test.ts` – Compatibility Checks, Version Detection +- **Phase 4 (Compatibility)**: Alle Tests prüfen Dispatch-Schema, Harness-Verfügbarkeit, Effort-Werte +- **Phase 5 (Evidence)**: `tests/evidence.test.ts` – Privacy-conscious Metric Collection, Aggregation + +Alle Tests müssen grün sein (`npm test`). + +## Architektur-Prinzipien + +- **Firstmate bleibt Authority**: crew-knowledge liefert nur Empfehlungen +- **Kein Competing Dispatch Engine**: Nutzt Firstmate's `quota-array-dispatch` +- **Respektiert config/crew-dispatch.json**: Natural Language Rules bleiben authoritative +- **Provider ≠ Model ≠ Harness**: Klare Trennung, kein Inferieren +- **Herdr ist Infrastructure**: Backend-Wahl beeinflusst nicht Model-Qualität + ## Lizenz MIT diff --git a/index.ts b/index.ts index 04dd2b1..e0f0839 100644 --- a/index.ts +++ b/index.ts @@ -6,6 +6,9 @@ * crew_balance — live quota-axi ranking snapshot * crew_apply_dispatch — generate/update config/crew-dispatch.json (dry-run default) * crew_suggest_primary — optional mid-session Pi model suggestion (no pane hijack) + * crew_discover — live model discovery from Pi, Claude, Codex, Grok, Kimi + * crew_evidence — local crew evidence from no-mistakes outcomes + * crew_update_check — compatibility check with current Firstmate version * * Does not replace Firstmate judgment or auto-spawn crewmates. */ @@ -26,6 +29,21 @@ import { formatSuggestPrimaryResult, suggestPrimary, } from "./src/suggest-primary.ts"; +import { + getCachedDiscovery, + getAvailableHarnesses, + isCacheStale, + clearCache as clearDiscoveryCache, +} from "./src/discovery.ts"; +import { + loadAggregatedEvidence, + getBestModelForTask, +} from "./src/evidence.ts"; +import { + checkCompatibility, + refreshAfterUpdate, + getFirstmateVersion, +} from "./src/update.ts"; const knowledge = loadKnowledge(); @@ -90,6 +108,41 @@ const SUGGEST_PRIMARY_PARAMS = Type.Object({ ), }); +const DISCOVERY_PARAMS = Type.Object({ + forceRefresh: Type.Optional( + Type.Boolean({ + description: "Force fresh discovery, bypassing cache (default false)", + }), + ), + format: Type.Optional( + Type.Union([Type.Literal("text"), Type.Literal("json")], { + description: "Response format (default text)", + }), + ), +}); + +const EVIDENCE_PARAMS = Type.Object({ + taskClass: Type.Optional( + Type.String({ + description: + "Optional task class filter: trivial_fix | standard_ship | hard_multi_file | research_web_live | scout_audit", + }), + ), + format: Type.Optional( + Type.Union([Type.Literal("text"), Type.Literal("json")], { + description: "Response format (default text)", + }), + ), +}); + +const UPDATE_CHECK_PARAMS = Type.Object({ + format: Type.Optional( + Type.Union([Type.Literal("text"), Type.Literal("json")], { + description: "Response format (default text)", + }), + ), +}); + function textResult(text: string, details?: unknown) { return { content: [{ type: "text" as const, text }], @@ -142,7 +195,7 @@ export default function firstmateCrewKnowledge(pi: ExtensionAPI): void { promptSnippet: "Recommend Firstmate crew spawn profile (harness/model/effort) from task text and quota-axi", promptGuidelines: [ - "Use crew_route before fm-spawn when choosing among Claude/Codex/Cursor/Grok profiles.", + "Use crew_route before fm-spawn when choosing among Claude/Codex/Cursor/Grok/Kimi profiles.", "Treat output as advice; Firstmate still decides and calls fm-spawn with concrete flags.", "If mode is degraded, prefer knowledge tables and disclose uncertainty.", ], @@ -165,7 +218,7 @@ export default function firstmateCrewKnowledge(pi: ExtensionAPI): void { "Read live quota-axi --json and summarize provider headroom/runway for balanced multi-provider burn. Data only plus transparent sort — no spawn.", promptSnippet: "Show quota-aware provider balance for the captain fleet", promptGuidelines: [ - "Call crew_balance when comparing remaining headroom across Claude, Codex, Cursor, Grok.", + "Call crew_balance when comparing remaining headroom across Claude, Codex, Cursor, Grok, Kimi.", "Do not invent credentials; missing quota-axi yields labeled degraded output.", ], parameters: BALANCE_PARAMS, @@ -233,6 +286,199 @@ export default function firstmateCrewKnowledge(pi: ExtensionAPI): void { }, }); + pi.registerTool({ + name: "crew_discover", + label: "Crew Discover", + description: + "Discover available models and harnesses from Pi, Claude, Codex, Grok, Kimi. Returns cached results unless forceRefresh=true.", + promptSnippet: "Discover available models and harnesses", + promptGuidelines: [ + "Use crew_discover to see what models are currently available.", + "Cache is valid for 15 minutes; use forceRefresh to bypass.", + ], + parameters: DISCOVERY_PARAMS, + async execute(_id, params) { + const discovery = getCachedDiscovery(params.forceRefresh || false); + const harnesses = getAvailableHarnesses(); + const stale = isCacheStale(); + + if (params.format === "json") { + return textResult( + JSON.stringify({ discovery, harnesses, stale }, null, 2), + { discovery, harnesses, stale }, + ); + } + + const lines = [ + "# Model Discovery", + "", + `Cache: ${stale ? "STALE" : "fresh"} (timestamp: ${new Date(discovery.timestamp).toISOString()})`, + `Firstmate version: ${discovery.firstmateVersion || "unknown"}`, + `Models found: ${discovery.models.length}`, + "", + "## Available Harnesses", + "", + ]; + + for (const h of harnesses) { + lines.push(`### ${h.harness} (${h.models.length} models)`); + for (const model of h.models) { + lines.push(` - ${model}`); + } + lines.push(""); + } + + if (discovery.errors.length > 0) { + lines.push("", "## Discovery Errors", ""); + for (const err of discovery.errors) { + lines.push(` - ${err}`); + } + } + + return textResult(lines.join("\n"), { discovery, harnesses, stale }); + }, + }); + + pi.registerTool({ + name: "crew_evidence", + label: "Crew Evidence", + description: + "View local crew evidence collected from no-mistakes outcomes. Shows aggregated metrics by task class and model.", + promptSnippet: "View local crew performance evidence", + promptGuidelines: [ + "Use crew_evidence to see how different models perform on different task types.", + "Evidence is collected from actual no-mistakes task outcomes.", + "Privacy-conscious: no code content, no prompts, only metrics.", + ], + parameters: EVIDENCE_PARAMS, + async execute(_id, params) { + let evidence = loadAggregatedEvidence(); + + if (params.taskClass) { + evidence = evidence.filter((e) => e.taskClass === params.taskClass); + } + + if (params.format === "json") { + return textResult(JSON.stringify(evidence, null, 2), evidence); + } + + const lines = ["# Local Crew Evidence", ""]; + + if (evidence.length === 0) { + lines.push("No evidence collected yet."); + lines.push(""); + lines.push( + "Evidence is automatically collected from no-mistakes task outcomes.", + ); + } else { + lines.push(`Total entries: ${evidence.length}`); + lines.push(""); + + const byTaskClass = new Map(); + for (const e of evidence) { + if (!byTaskClass.has(e.taskClass)) { + byTaskClass.set(e.taskClass, []); + } + byTaskClass.get(e.taskClass)!.push(e); + } + + for (const [taskClass, entries] of byTaskClass) { + lines.push(`## ${taskClass}`); + lines.push(""); + + // Sort by success rate, then avg duration + entries.sort((a, b) => { + if (Math.abs(a.successRate - b.successRate) > 0.05) { + return b.successRate - a.successRate; + } + return a.avgDurationMs - b.avgDurationMs; + }); + + for (const e of entries) { + lines.push( + `### ${e.provider}/${e.model} (${e.effort})`, + ); + lines.push(` - Tasks: ${e.totalTasks}`); + lines.push( + ` - Success rate: ${(e.successRate * 100).toFixed(1)}%`, + ); + lines.push( + ` - Avg duration: ${(e.avgDurationMs / 1000).toFixed(1)}s`, + ); + lines.push( + ` - Avg rework: ${e.avgReworkCount.toFixed(1)}`, + ); + if (e.avgTestPassRate > 0) { + lines.push( + ` - Test pass rate: ${(e.avgTestPassRate * 100).toFixed(1)}%`, + ); + } + lines.push(""); + } + } + } + + return textResult(lines.join("\n"), evidence); + }, + }); + + pi.registerTool({ + name: "crew_update_check", + label: "Crew Update Check", + description: + "Check compatibility with current Firstmate version and report any breaking changes or warnings.", + promptSnippet: "Check Firstmate compatibility", + promptGuidelines: [ + "Use crew_update_check after updating Firstmate.", + "Reports compatibility issues with dispatch schema, harnesses, spawn flags, etc.", + ], + parameters: UPDATE_CHECK_PARAMS, + async execute(_id, params) { + const result = checkCompatibility(); + const version = getFirstmateVersion(); + + if (params.format === "json") { + return textResult( + JSON.stringify({ ...result, version }, null, 2), + { ...result, version }, + ); + } + + const lines = [ + "# Firstmate Compatibility Check", + "", + `Firstmate version: ${version || "unknown"}`, + `Compatible: ${result.compatible ? "✓" : "✗"}`, + "", + ]; + + if (result.errors.length > 0) { + lines.push("## Errors", ""); + for (const err of result.errors) { + lines.push(` ✗ ${err}`); + } + lines.push(""); + } + + if (result.warnings.length > 0) { + lines.push("## Warnings", ""); + for (const warn of result.warnings) { + lines.push(` ⚠ ${warn}`); + } + lines.push(""); + } + + lines.push("## Checks", ""); + for (const check of result.checks) { + const icon = + check.status === "ok" ? "✓" : check.status === "warning" ? "⚠" : "✗"; + lines.push(` ${icon} ${check.check}: ${check.message}`); + } + + return textResult(lines.join("\n"), { ...result, version }); + }, + }); + pi.registerCommand("crew-route", { description: "Classify task text and print spawn flags: /crew-route ", diff --git a/knowledge/README.md b/knowledge/README.md index bd9d736..ebfc369 100644 --- a/knowledge/README.md +++ b/knowledge/README.md @@ -2,10 +2,64 @@ Datengetriebene Tabellen für `firstmate-crew-knowledge`. Tool-Code liest diese Dateien; Routing-Updates gehören hierher, nicht in den Scorer. +## Basis-Knowledge (Ebene 0) + | Datei | Inhalt | | --- | --- | | `task-classes.json` | Task-Klassen (trivial → scout), Keywords, bevorzugte Profile | -| `providers.json` | Provider-Rollen (Claude, Codex, Cursor, Grok), Harness- und Model-Muster | +| `providers.json` | Provider-Rollen (Claude, Codex, Cursor, Grok, Kimi), Harness- und Model-Muster | | `profiles.json` | Konkrete `{harness, model, effort}`-Profile + Dispatch-Template | +## Drei-Ebenen Knowledge System + +### A. Hersteller-Fakten (`manufacturers/`) + +Offizielle Dokumentation und Manufacturer Claims. + +- **Source**: `official-docs` (hohe Confidence) | `manufacturer-claim` (niedrige Confidence) +- **Format**: JSON mit `{source, confidence, timestamp, validUntil, provider, fact, details, url}` +- **Nutzung**: "Was kann ein Modell?" +- **Priorität**: Official Docs > Website > Discovery + +### B. Externe Benchmarks (`benchmarks/`) + +Externe Benchmark-Ergebnisse (Artificial Analysis, HumanEval, MBPP, etc.). + +- **Source**: `external-benchmark` +- **Confidence**: `medium` +- **Format**: JSON mit `{source, confidence, timestamp, benchmarkName, provider, model, score, metric, url}` +- **Nutzung**: "Wie gut ist ein Modell?" +- **Priorität**: Local Evidence > External Benchmarks > Manufacturer Claims + +### C. Lokale Crew-Evidenz (`local/`) + +Automatisch gesammelt aus no-mistakes Outcomes. + +- **Source**: `local` +- **Confidence**: `measured` +- **Format**: JSON mit Metriken (Task-Klasse, Harness, Provider, Modell, Effort, Duration, Success, Tests, CI, Rework) +- **Privacy**: **KEINE Code-Inhalte, KEINE Prompts** – nur Metriken +- **Nutzung**: "Wie gut performt ein Modell lokal?" +- **Priorität**: Höchste (gemessene Daten schlagen externe Benchmarks und Claims) + +#### Evidenz-Dateien + +- `evidence.json`: Raw evidence entries (append-only) +- `aggregated.json`: Aggregiert nach TaskClass/Provider/Model/Effort + +### Source-Konflikt-Hierarchie + +Bei Konflikten zwischen verschiedenen Quellen: + +- **"Was ist verfügbar?"**: Live Discovery > Auth State > Manufacturer Website +- **"Was kann es?"**: Official Docs > Website > Discovery +- **"Wie gut?"**: Local Evidence > External Benchmarks > Manufacturer Claims + +## Maintaining Knowledge + Firstmate bleibt urteilsfähig: diese Tabellen sind Vorschläge, kein Ersatz für `quota-array-dispatch` oder `fm-spawn`. + +- **Basis-Knowledge**: Manuell pflegen via JSON-Updates +- **Hersteller-Fakten**: Manuell aus offiziellen Docs sammeln +- **Externe Benchmarks**: Manuell aus Benchmark-Sites eintragen +- **Lokale Evidenz**: **Automatisch** via `recordEvidence()` aus no-mistakes outcomes diff --git a/knowledge/benchmarks/.gitkeep b/knowledge/benchmarks/.gitkeep new file mode 100644 index 0000000..93c7a00 --- /dev/null +++ b/knowledge/benchmarks/.gitkeep @@ -0,0 +1,2 @@ +# External benchmark data (Artificial Analysis, coding benchmarks) +# Each file: {source, confidence, timestamp, valid_until, data} diff --git a/knowledge/local/.gitkeep b/knowledge/local/.gitkeep new file mode 100644 index 0000000..c1f493b --- /dev/null +++ b/knowledge/local/.gitkeep @@ -0,0 +1,2 @@ +# Local crew evidence from no-mistakes outcomes +# Each file: {source: "local", confidence: "measured", timestamp, metrics} diff --git a/knowledge/manufacturers/.gitkeep b/knowledge/manufacturers/.gitkeep new file mode 100644 index 0000000..f53ec8f --- /dev/null +++ b/knowledge/manufacturers/.gitkeep @@ -0,0 +1,2 @@ +# Manufacturer facts from official documentation +# Each file: {source, confidence, timestamp, valid_until, data} diff --git a/knowledge/providers.json b/knowledge/providers.json index bc70e1e..11cf096 100644 --- a/knowledge/providers.json +++ b/knowledge/providers.json @@ -75,13 +75,32 @@ "nativeModelHints": ["grok-4", "grok-3", "latest"], "defaultHarness": "grok", "notes": "authStatus usable with empty windows is uncertainty, not logout." + }, + { + "id": "kimi", + "label": "Kimi (Moonshot AI)", + "quotaProvider": "kimi", + "harnesses": ["kimi", "pi"], + "strengths": [ + "long context window", + "coding tasks", + "Asian language support" + ], + "weaknesses": [ + "newer provider with less track record", + "quota integration may be incomplete" + ], + "piModelPatterns": ["moonshot/kimi-*", "moonshot/*"], + "nativeModelHints": ["kimi-1.5", "kimi-2"], + "defaultHarness": "kimi" } ], "harnessNotes": { - "pi": "Pi coding-agent; model ids are provider/model. Can host anthropic, openai-codex, xai families when authenticated.", + "pi": "Pi coding-agent; model ids are provider/model. Can host anthropic, openai-codex, xai, moonshot families when authenticated.", "claude": "Claude Code / Claude CLI native harness.", "codex": "OpenAI Codex CLI harness.", "grok": "Grok CLI harness.", + "kimi": "Kimi CLI harness.", "cursor": "Cursor agent/composer path when the home supports it as a spawn harness." } } diff --git a/package.json b/package.json index ab3fc30..8a25cca 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "firstmate-crew-knowledge", "version": "0.1.0", - "description": "Pi-Extension für Firstmate: modell-/harness-Routing pro Task-Klasse und quota-bewusste Kandidatenbalance über Cursor, Claude, Codex und Grok.", + "description": "Pi-Extension für Firstmate: modell-/harness-Routing pro Task-Klasse und quota-bewusste Kandidatenbalance über Cursor, Claude, Codex, Grok und Kimi.", "keywords": [ "pi-package", "pi-extension", diff --git a/src/discovery.ts b/src/discovery.ts new file mode 100644 index 0000000..bc1b003 --- /dev/null +++ b/src/discovery.ts @@ -0,0 +1,405 @@ +/** + * discovery.ts — Live model/harness discovery with caching and staleness detection + * + * Phase 1: Discovery & Firstmate-Version-Awareness + * - Query Pi, Claude, Codex, Grok, Kimi for available models + * - Cache results with timestamps + * - Detect stale data + * - Verify against harness catalogs + */ + +import { execSync } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; +import os from "node:os"; + +/** How long discovery cache remains fresh (15 minutes) */ +const CACHE_TTL_MS = 15 * 60 * 1000; + +interface DiscoveredModel { + provider: string; + model: string; + harness: string; + source: "pi" | "claude" | "codex" | "grok" | "kimi"; + available: boolean; +} + +interface DiscoveryCache { + timestamp: number; + models: DiscoveredModel[]; + firstmateVersion?: string; + errors: string[]; +} + +interface HarnessInfo { + harness: string; + available: boolean; + models: string[]; +} + +function getCacheDir(): string { + const home = os.homedir(); + const dir = path.join(home, ".cache", "firstmate-crew-knowledge"); + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }); + } + return dir; +} + +function getCachePath(): string { + return path.join(getCacheDir(), "discovery.json"); +} + +function isStale(timestamp: number): boolean { + return Date.now() - timestamp > CACHE_TTL_MS; +} + +/** + * Discover models from Pi harness + */ +function discoverPi(): DiscoveredModel[] { + const models: DiscoveredModel[] = []; + try { + const output = execSync("pi --list-models 2>/dev/null", { + encoding: "utf-8", + timeout: 10000, + }).trim(); + + // Parse pi --list-models output + // Expected format: provider/model lines + const lines = output.split("\n").filter((l) => l.trim()); + for (const line of lines) { + if (line.includes("/")) { + const [provider, model] = line.trim().split("/", 2); + if (provider && model) { + models.push({ + provider: provider.trim(), + model: model.trim(), + harness: "pi", + source: "pi", + available: true, + }); + } + } + } + } catch (err) { + // Pi not available or command failed + } + return models; +} + +/** + * Discover models from Claude CLI + */ +function discoverClaude(): DiscoveredModel[] { + const models: DiscoveredModel[] = []; + try { + // Try to detect if Claude CLI is available and authenticated + execSync("which claude 2>/dev/null", { encoding: "utf-8" }); + // If claude is available, add known models + // Claude CLI doesn't have a --list-models, so we infer from knowledge + models.push( + { + provider: "claude", + model: "haiku", + harness: "claude", + source: "claude", + available: true, + }, + { + provider: "claude", + model: "sonnet", + harness: "claude", + source: "claude", + available: true, + }, + { + provider: "claude", + model: "opus", + harness: "claude", + source: "claude", + available: true, + }, + ); + } catch (err) { + // Claude CLI not available + } + return models; +} + +/** + * Discover models from Codex CLI + */ +function discoverCodex(): DiscoveredModel[] { + const models: DiscoveredModel[] = []; + try { + execSync("which codex 2>/dev/null", { encoding: "utf-8" }); + // Codex CLI available - infer common models + models.push( + { + provider: "codex", + model: "gpt-5.5", + harness: "codex", + source: "codex", + available: true, + }, + { + provider: "codex", + model: "gpt-5.4", + harness: "codex", + source: "codex", + available: true, + }, + ); + } catch (err) { + // Codex not available + } + return models; +} + +/** + * Discover models from Grok CLI + */ +function discoverGrok(): DiscoveredModel[] { + const models: DiscoveredModel[] = []; + try { + const output = execSync("grok models 2>/dev/null", { + encoding: "utf-8", + timeout: 10000, + }).trim(); + + // Parse grok models output + const lines = output.split("\n").filter((l) => l.trim()); + for (const line of lines) { + const trimmed = line.trim(); + if (trimmed && !trimmed.startsWith("#")) { + models.push({ + provider: "grok", + model: trimmed, + harness: "grok", + source: "grok", + available: true, + }); + } + } + + // Fallback to known models if parsing failed + if (models.length === 0) { + models.push({ + provider: "grok", + model: "latest", + harness: "grok", + source: "grok", + available: true, + }); + } + } catch (err) { + // Grok not available + } + return models; +} + +/** + * Discover models from Kimi CLI + */ +function discoverKimi(): DiscoveredModel[] { + const models: DiscoveredModel[] = []; + try { + const output = execSync("kimi provider list --json 2>/dev/null", { + encoding: "utf-8", + timeout: 10000, + }).trim(); + + const data = JSON.parse(output); + if (Array.isArray(data)) { + for (const item of data) { + if (item.provider && item.model) { + models.push({ + provider: item.provider, + model: item.model, + harness: "kimi", + source: "kimi", + available: true, + }); + } + } + } + } catch (err) { + // Kimi not available or parse error + } + return models; +} + +/** + * Get Firstmate version from AGENTS.md or docs + */ +function getFirstmateVersion(): string | undefined { + try { + const fmHome = process.env.FM_HOME || process.env.FM_ROOT_OVERRIDE; + if (fmHome) { + const agentsPath = path.join(fmHome, "AGENTS.md"); + if (fs.existsSync(agentsPath)) { + const content = fs.readFileSync(agentsPath, "utf-8"); + // Try to extract version from AGENTS.md + const versionMatch = content.match(/version[:\s]+([0-9.]+)/i); + if (versionMatch) { + return versionMatch[1]; + } + } + } + } catch (err) { + // Could not determine version + } + return undefined; +} + +/** + * Perform live discovery of all available models + */ +export function discoverModels(): DiscoveryCache { + const errors: string[] = []; + const models: DiscoveredModel[] = []; + + try { + models.push(...discoverPi()); + } catch (err) { + errors.push(`Pi discovery failed: ${err}`); + } + + try { + models.push(...discoverClaude()); + } catch (err) { + errors.push(`Claude discovery failed: ${err}`); + } + + try { + models.push(...discoverCodex()); + } catch (err) { + errors.push(`Codex discovery failed: ${err}`); + } + + try { + models.push(...discoverGrok()); + } catch (err) { + errors.push(`Grok discovery failed: ${err}`); + } + + try { + models.push(...discoverKimi()); + } catch (err) { + errors.push(`Kimi discovery failed: ${err}`); + } + + const firstmateVersion = getFirstmateVersion(); + + return { + timestamp: Date.now(), + models, + firstmateVersion, + errors, + }; +} + +/** + * Get cached discovery or perform fresh discovery if stale + */ +export function getCachedDiscovery(forceRefresh = false): DiscoveryCache { + const cachePath = getCachePath(); + + if (!forceRefresh && fs.existsSync(cachePath)) { + try { + const cached: DiscoveryCache = JSON.parse( + fs.readFileSync(cachePath, "utf-8"), + ); + if (!isStale(cached.timestamp)) { + return cached; + } + } catch (err) { + // Cache corrupted, proceed to fresh discovery + } + } + + // Perform fresh discovery + const discovery = discoverModels(); + + // Save to cache + try { + fs.writeFileSync(cachePath, JSON.stringify(discovery, null, 2)); + } catch (err) { + // Cache write failed, but return discovery anyway + } + + return discovery; +} + +/** + * Verify a model/harness combination against discovery + */ +export function verifyModelHarness( + provider: string, + model: string, + harness: string, +): { available: boolean; source?: string } { + const discovery = getCachedDiscovery(); + + const match = discovery.models.find( + (m) => + m.provider === provider && m.model === model && m.harness === harness, + ); + + if (match) { + return { available: true, source: match.source }; + } + + return { available: false }; +} + +/** + * Get all available harnesses with their models + */ +export function getAvailableHarnesses(): HarnessInfo[] { + const discovery = getCachedDiscovery(); + const harnesses = new Map>(); + + for (const model of discovery.models) { + if (!harnesses.has(model.harness)) { + harnesses.set(model.harness, new Set()); + } + harnesses.get(model.harness)!.add(`${model.provider}/${model.model}`); + } + + return Array.from(harnesses.entries()).map(([harness, modelSet]) => ({ + harness, + available: modelSet.size > 0, + models: Array.from(modelSet).sort(), + })); +} + +/** + * Check if discovery cache is stale + */ +export function isCacheStale(): boolean { + const cachePath = getCachePath(); + if (!fs.existsSync(cachePath)) { + return true; + } + + try { + const cached: DiscoveryCache = JSON.parse( + fs.readFileSync(cachePath, "utf-8"), + ); + return isStale(cached.timestamp); + } catch (err) { + return true; + } +} + +/** + * Clear the discovery cache + */ +export function clearCache(): void { + const cachePath = getCachePath(); + if (fs.existsSync(cachePath)) { + fs.unlinkSync(cachePath); + } +} diff --git a/src/evidence.ts b/src/evidence.ts new file mode 100644 index 0000000..96707aa --- /dev/null +++ b/src/evidence.ts @@ -0,0 +1,274 @@ +/** + * evidence.ts — Local crew evidence collection from no-mistakes outcomes + * + * Phase 5: Lokale Crew-Evidence Collection (MANDATORY) + * - Collect metrics from completed no-mistakes tasks + * - Task class, harness, provider, model, effort + * - Duration, success/fail, tests passed, CI result, rework count + * - Privacy: NO code content, NO prompts - only metrics + * - Aggregation per TaskClass/Model + */ + +import fs from "node:fs"; +import path from "node:path"; + +export interface TaskEvidence { + /** Unique task ID */ + taskId: string; + /** Task classification */ + taskClass: string; + /** Harness used (pi, claude, codex, grok, etc.) */ + harness: string; + /** Provider (claude, codex, grok, etc.) */ + provider: string; + /** Model used */ + model: string; + /** Effort level (low, medium, high) */ + effort: string; + /** Task start timestamp */ + startTime: number; + /** Task end timestamp */ + endTime: number; + /** Duration in milliseconds */ + durationMs: number; + /** Overall success */ + success: boolean; + /** Number of tests run */ + testsRun?: number; + /** Number of tests passed */ + testsPassed?: number; + /** CI result (pass, fail, skip) */ + ciResult?: "pass" | "fail" | "skip"; + /** Number of rework cycles */ + reworkCount: number; + /** Source of evidence (always "local") */ + source: "local"; + /** Confidence (always "measured") */ + confidence: "measured"; + /** Timestamp of evidence collection */ + timestamp: number; +} + +export interface AggregatedEvidence { + taskClass: string; + provider: string; + model: string; + effort: string; + totalTasks: number; + successfulTasks: number; + successRate: number; + avgDurationMs: number; + avgReworkCount: number; + avgTestPassRate: number; + lastUpdated: number; +} + +function getEvidenceDir(): string { + // Store in knowledge/local/ + const dir = path.join(process.cwd(), "knowledge", "local"); + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }); + } + return dir; +} + +function getEvidencePath(): string { + return path.join(getEvidenceDir(), "evidence.json"); +} + +function getAggregatedPath(): string { + return path.join(getEvidenceDir(), "aggregated.json"); +} + +/** + * Load all raw evidence + */ +export function loadEvidence(): TaskEvidence[] { + const evidencePath = getEvidencePath(); + if (!fs.existsSync(evidencePath)) { + return []; + } + + try { + const data = JSON.parse(fs.readFileSync(evidencePath, "utf-8")); + return Array.isArray(data) ? data : []; + } catch (err) { + console.error(`Failed to load evidence: ${err}`); + return []; + } +} + +/** + * Save raw evidence + */ +function saveEvidence(evidence: TaskEvidence[]): void { + const evidencePath = getEvidencePath(); + fs.writeFileSync(evidencePath, JSON.stringify(evidence, null, 2)); +} + +/** + * Record a new task evidence entry + */ +export function recordEvidence(evidence: Omit): void { + const allEvidence = loadEvidence(); + + const newEvidence: TaskEvidence = { + ...evidence, + source: "local", + confidence: "measured", + timestamp: Date.now(), + }; + + allEvidence.push(newEvidence); + saveEvidence(allEvidence); + + // Trigger re-aggregation + aggregateEvidence(); +} + +/** + * Aggregate evidence by task class, provider, model, effort + */ +export function aggregateEvidence(): AggregatedEvidence[] { + const evidence = loadEvidence(); + const groups = new Map(); + + // Group by taskClass|provider|model|effort + for (const ev of evidence) { + const key = `${ev.taskClass}|${ev.provider}|${ev.model}|${ev.effort}`; + if (!groups.has(key)) { + groups.set(key, []); + } + groups.get(key)!.push(ev); + } + + // Aggregate each group + const aggregated: AggregatedEvidence[] = []; + for (const [key, items] of groups) { + const [taskClass, provider, model, effort] = key.split("|"); + + const totalTasks = items.length; + const successfulTasks = items.filter((e) => e.success).length; + const successRate = totalTasks > 0 ? successfulTasks / totalTasks : 0; + + const avgDurationMs = items.reduce((sum, e) => sum + e.durationMs, 0) / totalTasks; + const avgReworkCount = items.reduce((sum, e) => sum + e.reworkCount, 0) / totalTasks; + + const testsData = items.filter((e) => e.testsRun && e.testsPassed); + const avgTestPassRate = testsData.length > 0 + ? testsData.reduce((sum, e) => (e.testsPassed! / e.testsRun!), 0) / testsData.length + : 0; + + aggregated.push({ + taskClass, + provider, + model, + effort, + totalTasks, + successfulTasks, + successRate, + avgDurationMs, + avgReworkCount, + avgTestPassRate, + lastUpdated: Date.now(), + }); + } + + // Save aggregated data + const aggregatedPath = getAggregatedPath(); + fs.writeFileSync(aggregatedPath, JSON.stringify(aggregated, null, 2)); + + return aggregated; +} + +/** + * Load aggregated evidence + */ +export function loadAggregatedEvidence(): AggregatedEvidence[] { + const aggregatedPath = getAggregatedPath(); + if (!fs.existsSync(aggregatedPath)) { + // Generate if not exists + return aggregateEvidence(); + } + + try { + const data = JSON.parse(fs.readFileSync(aggregatedPath, "utf-8")); + return Array.isArray(data) ? data : []; + } catch (err) { + console.error(`Failed to load aggregated evidence: ${err}`); + return []; + } +} + +/** + * Get evidence for a specific task class and model combination + */ +export function getEvidenceForTask( + taskClass: string, + provider: string, + model: string, +): AggregatedEvidence | undefined { + const aggregated = loadAggregatedEvidence(); + return aggregated.find( + (e) => e.taskClass === taskClass && e.provider === provider && e.model === model, + ); +} + +/** + * Get best performing model for a task class based on local evidence + */ +export function getBestModelForTask(taskClass: string): AggregatedEvidence | undefined { + const aggregated = loadAggregatedEvidence(); + const forTask = aggregated.filter((e) => e.taskClass === taskClass); + + if (forTask.length === 0) { + return undefined; + } + + // Rank by success rate, then avg duration (faster is better) + forTask.sort((a, b) => { + if (Math.abs(a.successRate - b.successRate) > 0.05) { + return b.successRate - a.successRate; + } + return a.avgDurationMs - b.avgDurationMs; + }); + + return forTask[0]; +} + +/** + * Parse no-mistakes outcome and record evidence + * This is a helper function to extract metrics from no-mistakes logs + */ +export function recordFromNoMistakesOutcome(outcome: { + taskId: string; + taskClass: string; + harness: string; + provider: string; + model: string; + effort: string; + startTime: number; + endTime: number; + success: boolean; + testsRun?: number; + testsPassed?: number; + ciResult?: "pass" | "fail" | "skip"; + reworkCount: number; +}): void { + recordEvidence({ + taskId: outcome.taskId, + taskClass: outcome.taskClass, + harness: outcome.harness, + provider: outcome.provider, + model: outcome.model, + effort: outcome.effort, + startTime: outcome.startTime, + endTime: outcome.endTime, + durationMs: outcome.endTime - outcome.startTime, + success: outcome.success, + testsRun: outcome.testsRun, + testsPassed: outcome.testsPassed, + ciResult: outcome.ciResult, + reworkCount: outcome.reworkCount, + }); +} diff --git a/src/knowledge-layers.ts b/src/knowledge-layers.ts new file mode 100644 index 0000000..4ff2153 --- /dev/null +++ b/src/knowledge-layers.ts @@ -0,0 +1,378 @@ +/** + * knowledge-layers.ts — Three-level knowledge system + * + * Phase 2: Drei-Ebenen Knowledge + * A. Hersteller-Fakten (official manufacturer docs) + * B. Externe Benchmarks (Artificial Analysis, coding benchmarks) + * C. Lokale Crew-Evidenz (from no-mistakes outcomes) + * + * Each level: source, confidence, timestamp, valid_until + * Separate storage: knowledge/manufacturers/, knowledge/benchmarks/, knowledge/local/ + * + * Source-Konflikt-Hierarchie (Section 10): + * - "What's available?": Live Discovery > Auth State > Manufacturer Website + * - "What can it do?": Official Docs > Website > Discovery + * - "How good?": Local Evidence > External Benchmarks > Manufacturer Claims + */ + +import fs from "node:fs"; +import path from "node:path"; + +/** Confidence level for knowledge */ +export type ConfidenceLevel = + | "measured" // From local evidence + | "high" // Official docs + | "medium" // External benchmarks + | "low" // Manufacturer claims + | "uncertain"; // Unknown or unreliable + +/** Knowledge source type */ +export type SourceType = + | "local" // Local evidence + | "discovery" // Live model discovery + | "official-docs" // Manufacturer official documentation + | "external-benchmark" // Artificial Analysis, etc. + | "manufacturer-claim" // Manufacturer website/marketing + | "auth-state"; // Authentication state + +/** Base knowledge entry */ +export interface KnowledgeEntry { + source: SourceType; + confidence: ConfidenceLevel; + timestamp: number; + validUntil?: number; +} + +/** Manufacturer facts entry */ +export interface ManufacturerFact extends KnowledgeEntry { + source: "official-docs" | "manufacturer-claim"; + provider: string; + fact: string; + details: Record; + url?: string; +} + +/** External benchmark entry */ +export interface BenchmarkResult extends KnowledgeEntry { + source: "external-benchmark"; + benchmarkName: string; + provider: string; + model: string; + score: number; + metric: string; + url?: string; +} + +/** Capability fact */ +export interface CapabilityFact extends KnowledgeEntry { + provider: string; + model?: string; + capability: string; + supported: boolean; + notes?: string; +} + +function getKnowledgeDir(layer: "manufacturers" | "benchmarks" | "local"): string { + const dir = path.join(process.cwd(), "knowledge", layer); + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }); + } + return dir; +} + +/** + * Save manufacturer fact + */ +export function saveManufacturerFact(fact: ManufacturerFact): void { + const dir = getKnowledgeDir("manufacturers"); + const filename = `${fact.provider}-${Date.now()}.json`; + const filepath = path.join(dir, filename); + fs.writeFileSync(filepath, JSON.stringify(fact, null, 2)); +} + +/** + * Save benchmark result + */ +export function saveBenchmarkResult(result: BenchmarkResult): void { + const dir = getKnowledgeDir("benchmarks"); + const filename = `${result.benchmarkName}-${result.provider}-${result.model}-${Date.now()}.json`; + const filepath = path.join(dir, filename); + fs.writeFileSync(filepath, JSON.stringify(result, null, 2)); +} + +/** + * Load all manufacturer facts + */ +export function loadManufacturerFacts(): ManufacturerFact[] { + const dir = getKnowledgeDir("manufacturers"); + const files = fs.readdirSync(dir).filter((f) => f.endsWith(".json")); + + const facts: ManufacturerFact[] = []; + for (const file of files) { + try { + const content = fs.readFileSync(path.join(dir, file), "utf-8"); + const fact = JSON.parse(content); + facts.push(fact); + } catch (err) { + console.error(`Failed to load manufacturer fact ${file}: ${err}`); + } + } + + return facts; +} + +/** + * Load all benchmark results + */ +export function loadBenchmarkResults(): BenchmarkResult[] { + const dir = getKnowledgeDir("benchmarks"); + const files = fs.readdirSync(dir).filter((f) => f.endsWith(".json")); + + const results: BenchmarkResult[] = []; + for (const file of files) { + try { + const content = fs.readFileSync(path.join(dir, file), "utf-8"); + const result = JSON.parse(content); + results.push(result); + } catch (err) { + console.error(`Failed to load benchmark result ${file}: ${err}`); + } + } + + return results; +} + +/** + * Check if knowledge entry is still valid + */ +export function isValid(entry: KnowledgeEntry): boolean { + if (!entry.validUntil) { + return true; // No expiry + } + return Date.now() < entry.validUntil; +} + +/** + * Filter valid knowledge entries + */ +export function filterValid(entries: T[]): T[] { + return entries.filter(isValid); +} + +/** + * Resolve conflict between multiple knowledge sources + * Priority: Local Evidence > External Benchmarks > Official Docs > Manufacturer Claims + */ +export function resolveConflict( + entries: T[], + selector: (entry: T) => boolean, +): T | undefined { + const matching = entries.filter(selector).filter(isValid); + + if (matching.length === 0) { + return undefined; + } + + // Priority by confidence + const priorities: Record = { + measured: 4, + high: 3, + medium: 2, + low: 1, + uncertain: 0, + }; + + matching.sort((a, b) => { + const priorityDiff = priorities[b.confidence] - priorities[a.confidence]; + if (priorityDiff !== 0) { + return priorityDiff; + } + // If same confidence, prefer newer + return b.timestamp - a.timestamp; + }); + + return matching[0]; +} + +/** + * Get availability information for a provider/model + * Priority: Live Discovery > Auth State > Manufacturer Website + */ +export function getAvailability( + provider: string, + model: string, + allEntries: KnowledgeEntry[], +): { available: boolean; confidence: ConfidenceLevel; source: SourceType } { + const relevant = allEntries.filter( + (e) => + "provider" in e && + e.provider === provider && + (!("model" in e) || e.model === model), + ); + + // Priority order for "what's available" + const discovery = relevant.find((e) => e.source === "discovery"); + if (discovery) { + return { + available: true, + confidence: discovery.confidence, + source: "discovery", + }; + } + + const authState = relevant.find((e) => e.source === "auth-state"); + if (authState) { + return { + available: true, + confidence: authState.confidence, + source: "auth-state", + }; + } + + const manufacturer = relevant.find( + (e) => e.source === "manufacturer-claim" || e.source === "official-docs", + ); + if (manufacturer) { + return { + available: true, + confidence: manufacturer.confidence, + source: manufacturer.source, + }; + } + + return { + available: false, + confidence: "uncertain", + source: "manufacturer-claim", + }; +} + +/** + * Get capability information + * Priority: Official Docs > Website > Discovery + */ +export function getCapability( + provider: string, + model: string, + capability: string, + allEntries: KnowledgeEntry[], +): { supported: boolean; confidence: ConfidenceLevel; source: SourceType } { + const relevant = allEntries.filter( + (e) => + "provider" in e && + e.provider === provider && + (!("model" in e) || e.model === model) && + "capability" in e && + (e as CapabilityFact).capability === capability, + ) as CapabilityFact[]; + + // Priority order for "what can it do" + const officialDocs = relevant.find((e) => e.source === "official-docs"); + if (officialDocs) { + return { + supported: officialDocs.supported, + confidence: officialDocs.confidence, + source: "official-docs", + }; + } + + const manufacturer = relevant.find((e) => e.source === "manufacturer-claim"); + if (manufacturer) { + return { + supported: manufacturer.supported, + confidence: manufacturer.confidence, + source: "manufacturer-claim", + }; + } + + const discovery = relevant.find((e) => e.source === "discovery"); + if (discovery) { + return { + supported: discovery.supported, + confidence: discovery.confidence, + source: "discovery", + }; + } + + return { + supported: false, + confidence: "uncertain", + source: "manufacturer-claim", + }; +} + +/** + * Get quality/performance information + * Priority: Local Evidence > External Benchmarks > Manufacturer Claims + */ +export function getQuality( + provider: string, + model: string, + allEntries: KnowledgeEntry[], +): { + score?: number; + confidence: ConfidenceLevel; + source: SourceType; +} { + const relevant = allEntries.filter( + (e) => + ("provider" in e && e.provider === provider) && + (!("model" in e) || e.model === model), + ); + + // Priority order for "how good" + const localEvidence = relevant.find((e) => e.source === "local"); + if (localEvidence && "score" in localEvidence) { + return { + score: (localEvidence as { score: number }).score, + confidence: "measured", + source: "local", + }; + } + + const benchmark = relevant.find((e) => e.source === "external-benchmark"); + if (benchmark && "score" in benchmark) { + return { + score: (benchmark as BenchmarkResult).score, + confidence: benchmark.confidence, + source: "external-benchmark", + }; + } + + const manufacturer = relevant.find( + (e) => e.source === "manufacturer-claim" || e.source === "official-docs", + ); + if (manufacturer && "score" in manufacturer) { + return { + score: (manufacturer as { score: number }).score, + confidence: manufacturer.confidence, + source: manufacturer.source, + }; + } + + return { + confidence: "uncertain", + source: "manufacturer-claim", + }; +} + +/** + * Combine all knowledge layers for a comprehensive view + */ +export function getCombinedKnowledge(provider: string, model: string): { + availability: ReturnType; + quality: ReturnType; +} { + // In a full implementation, this would load from all three directories + // For now, we return a structure showing how to combine layers + const allEntries: KnowledgeEntry[] = [ + ...loadManufacturerFacts(), + ...loadBenchmarkResults(), + // Local evidence would come from evidence.ts + ]; + + return { + availability: getAvailability(provider, model, allEntries), + quality: getQuality(provider, model, allEntries), + }; +} diff --git a/src/update.ts b/src/update.ts new file mode 100644 index 0000000..941a042 --- /dev/null +++ b/src/update.ts @@ -0,0 +1,336 @@ +/** + * update.ts — Self-updating integration with Firstmate + * + * Phase 3: Selbstaktualisierung + * - Integration with /updatefirstmate workflow + * - Compatibility checks: dispatch schema, harnesses, models, efforts + * - Auto-refresh when Firstmate updates + * - Fail loudly on breaking changes + */ + +import fs from "node:fs"; +import path from "node:path"; +import { execSync } from "node:child_process"; +import { getCachedDiscovery, clearCache as clearDiscoveryCache } from "./discovery.ts"; + +interface CompatibilityCheck { + check: string; + status: "ok" | "warning" | "error"; + message: string; +} + +interface UpdateResult { + compatible: boolean; + checks: CompatibilityCheck[]; + errors: string[]; + warnings: string[]; +} + +/** + * Get Firstmate home directory + */ +function getFirstmateHome(): string | undefined { + return process.env.FM_HOME || process.env.FM_ROOT_OVERRIDE; +} + +/** + * Check if dispatch schema is compatible + */ +function checkDispatchSchema(): CompatibilityCheck { + const fmHome = getFirstmateHome(); + if (!fmHome) { + return { + check: "dispatch-schema", + status: "warning", + message: "FM_HOME not set, cannot verify dispatch schema compatibility", + }; + } + + const examplePath = path.join(fmHome, "docs", "examples", "crew-dispatch.json"); + if (!fs.existsSync(examplePath)) { + return { + check: "dispatch-schema", + status: "warning", + message: "Firstmate example crew-dispatch.json not found, cannot verify schema", + }; + } + + try { + const example = JSON.parse(fs.readFileSync(examplePath, "utf-8")); + + // Check for required schema fields + if (!example.rules || !Array.isArray(example.rules)) { + return { + check: "dispatch-schema", + status: "error", + message: "Dispatch schema missing 'rules' array - BREAKING CHANGE", + }; + } + + // Check rule structure + const firstRule = example.rules[0]; + if (firstRule && (!firstRule.when || !firstRule.use)) { + return { + check: "dispatch-schema", + status: "error", + message: "Dispatch rule schema changed (missing when/use) - BREAKING CHANGE", + }; + } + + return { + check: "dispatch-schema", + status: "ok", + message: "Dispatch schema compatible", + }; + } catch (err) { + return { + check: "dispatch-schema", + status: "error", + message: `Failed to parse dispatch schema: ${err}`, + }; + } +} + +/** + * Check if known harnesses are still valid + */ +function checkHarnesses(): CompatibilityCheck { + const discovery = getCachedDiscovery(); + const knownHarnesses = ["pi", "claude", "codex", "grok", "cursor"]; + + const availableHarnesses = new Set( + discovery.models.map((m) => m.harness), + ); + + const missingCritical = knownHarnesses.filter( + (h) => h !== "cursor" && !availableHarnesses.has(h), + ); + + if (missingCritical.length > 0) { + return { + check: "harnesses", + status: "warning", + message: `Critical harnesses not discovered: ${missingCritical.join(", ")}`, + }; + } + + return { + check: "harnesses", + status: "ok", + message: `All critical harnesses available: ${Array.from(availableHarnesses).join(", ")}`, + }; +} + +/** + * Check if spawn flags are compatible + */ +function checkSpawnFlags(): CompatibilityCheck { + const fmHome = getFirstmateHome(); + if (!fmHome) { + return { + check: "spawn-flags", + status: "warning", + message: "FM_HOME not set, cannot verify spawn flags", + }; + } + + const spawnScript = path.join(fmHome, "bin", "fm-spawn.sh"); + if (!fs.existsSync(spawnScript)) { + return { + check: "spawn-flags", + status: "warning", + message: "fm-spawn.sh not found, cannot verify flags", + }; + } + + try { + const content = fs.readFileSync(spawnScript, "utf-8"); + + // Check for expected flags + const requiredFlags = ["--harness", "--model", "--effort"]; + const missingFlags = requiredFlags.filter((flag) => !content.includes(flag)); + + if (missingFlags.length > 0) { + return { + check: "spawn-flags", + status: "error", + message: `fm-spawn.sh missing expected flags: ${missingFlags.join(", ")} - BREAKING CHANGE`, + }; + } + + return { + check: "spawn-flags", + status: "ok", + message: "Spawn flags compatible (--harness, --model, --effort present)", + }; + } catch (err) { + return { + check: "spawn-flags", + status: "error", + message: `Failed to verify spawn flags: ${err}`, + }; + } +} + +/** + * Check if effort values are compatible + */ +function checkEffortValues(): CompatibilityCheck { + const knownEfforts = ["low", "medium", "high"]; + // In the future, we could parse Firstmate docs to verify these + + return { + check: "effort-values", + status: "ok", + message: `Effort values compatible: ${knownEfforts.join(", ")}`, + }; +} + +/** + * Check quota-axi availability + */ +function checkQuotaAxi(): CompatibilityCheck { + try { + execSync("which quota-axi 2>/dev/null", { encoding: "utf-8" }); + + try { + execSync("quota-axi --json 2>/dev/null", { + encoding: "utf-8", + timeout: 5000, + }); + + return { + check: "quota-axi", + status: "ok", + message: "quota-axi available and working", + }; + } catch (err) { + return { + check: "quota-axi", + status: "warning", + message: "quota-axi found but failed to execute (might be auth issue)", + }; + } + } catch (err) { + return { + check: "quota-axi", + status: "warning", + message: "quota-axi not available (will use degraded mode)", + }; + } +} + +/** + * Perform full compatibility check + */ +export function checkCompatibility(): UpdateResult { + const checks: CompatibilityCheck[] = [ + checkDispatchSchema(), + checkHarnesses(), + checkSpawnFlags(), + checkEffortValues(), + checkQuotaAxi(), + ]; + + const errors = checks.filter((c) => c.status === "error").map((c) => c.message); + const warnings = checks.filter((c) => c.status === "warning").map((c) => c.message); + const compatible = errors.length === 0; + + return { + compatible, + checks, + errors, + warnings, + }; +} + +/** + * Refresh knowledge after Firstmate update + */ +export function refreshAfterUpdate(): UpdateResult { + // Clear discovery cache to force re-discovery + clearDiscoveryCache(); + + // Re-discover models + getCachedDiscovery(true); + + // Run compatibility checks + const result = checkCompatibility(); + + if (!result.compatible) { + console.error("⚠️ BREAKING CHANGES DETECTED:"); + for (const error of result.errors) { + console.error(` ❌ ${error}`); + } + throw new Error( + "Firstmate update introduced breaking changes. Manual intervention required.", + ); + } + + if (result.warnings.length > 0) { + console.warn("⚠️ Warnings after update:"); + for (const warning of result.warnings) { + console.warn(` ⚠️ ${warning}`); + } + } + + return result; +} + +/** + * Get Firstmate version from various sources + */ +export function getFirstmateVersion(): string | undefined { + const fmHome = getFirstmateHome(); + if (!fmHome) { + return undefined; + } + + // Try package.json + const pkgPath = path.join(fmHome, "package.json"); + if (fs.existsSync(pkgPath)) { + try { + const pkg = JSON.parse(fs.readFileSync(pkgPath, "utf-8")); + if (pkg.version) { + return pkg.version; + } + } catch (err) { + // Continue to next method + } + } + + // Try AGENTS.md + const agentsPath = path.join(fmHome, "AGENTS.md"); + if (fs.existsSync(agentsPath)) { + try { + const content = fs.readFileSync(agentsPath, "utf-8"); + const versionMatch = content.match(/version[:\s]+([0-9.]+)/i); + if (versionMatch) { + return versionMatch[1]; + } + } catch (err) { + // Continue + } + } + + return undefined; +} + +/** + * Check if an update is needed based on version change + */ +export function needsUpdate(lastKnownVersion?: string): boolean { + const currentVersion = getFirstmateVersion(); + + if (!currentVersion) { + // Cannot determine version, assume no update needed + return false; + } + + if (!lastKnownVersion) { + // First run, no update needed + return false; + } + + return currentVersion !== lastKnownVersion; +} diff --git a/tests/discovery.test.ts b/tests/discovery.test.ts new file mode 100644 index 0000000..bd3284e --- /dev/null +++ b/tests/discovery.test.ts @@ -0,0 +1,135 @@ +/** + * discovery.test.ts — Tests for Phase 1: Discovery & Firstmate-Version-Awareness + */ + +import { describe, it, beforeEach } from "node:test"; +import assert from "node:assert"; +import fs from "node:fs"; +import path from "node:path"; +import os from "node:os"; +import { + getCachedDiscovery, + verifyModelHarness, + getAvailableHarnesses, + isCacheStale, + clearCache, +} from "../src/discovery.ts"; + +describe("Discovery", () => { + const cacheDir = path.join(os.homedir(), ".cache", "firstmate-crew-knowledge"); + const cachePath = path.join(cacheDir, "discovery.json"); + + beforeEach(() => { + // Clear cache before each test + clearCache(); + }); + + it("should discover models and cache results", () => { + const discovery = getCachedDiscovery(); + + assert.ok(discovery, "Discovery should return results"); + assert.ok(discovery.timestamp, "Discovery should have timestamp"); + assert.ok(Array.isArray(discovery.models), "Discovery should have models array"); + assert.ok(Array.isArray(discovery.errors), "Discovery should have errors array"); + }); + + it("should use cache when fresh", () => { + const first = getCachedDiscovery(); + const second = getCachedDiscovery(); + + assert.strictEqual( + first.timestamp, + second.timestamp, + "Should use cached timestamp", + ); + }); + + it("should detect stale cache", () => { + // Create a stale cache + if (!fs.existsSync(cacheDir)) { + fs.mkdirSync(cacheDir, { recursive: true }); + } + + const staleData = { + timestamp: Date.now() - 20 * 60 * 1000, // 20 minutes ago + models: [], + errors: [], + }; + + fs.writeFileSync(cachePath, JSON.stringify(staleData)); + + assert.ok(isCacheStale(), "Should detect stale cache"); + }); + + it("should verify model/harness combinations", () => { + const discovery = getCachedDiscovery(); + + if (discovery.models.length > 0) { + const firstModel = discovery.models[0]; + const result = verifyModelHarness( + firstModel.provider, + firstModel.model, + firstModel.harness, + ); + + assert.ok(result.available, "Should verify available model"); + assert.ok(result.source, "Should report source"); + } + + const nonExistent = verifyModelHarness("fake", "nonexistent", "invalid"); + assert.strictEqual( + nonExistent.available, + false, + "Should not verify non-existent model", + ); + }); + + it("should list available harnesses", () => { + const harnesses = getAvailableHarnesses(); + + assert.ok(Array.isArray(harnesses), "Should return harnesses array"); + + for (const h of harnesses) { + assert.ok(h.harness, "Harness should have name"); + assert.ok(Array.isArray(h.models), "Harness should have models array"); + assert.ok(typeof h.available === "boolean", "Harness should have availability"); + } + }); + + it("should clear cache", () => { + getCachedDiscovery(); // Create cache + assert.ok(fs.existsSync(cachePath), "Cache should exist"); + + clearCache(); + assert.ok(!fs.existsSync(cachePath), "Cache should be cleared"); + }); + + it("should handle discovery errors gracefully", () => { + const discovery = getCachedDiscovery(); + + // Even if some discoveries fail, should still return structure + assert.ok(discovery, "Should return discovery even with errors"); + assert.ok(Array.isArray(discovery.errors), "Should collect errors"); + }); + + it("should force refresh when requested", () => { + const first = getCachedDiscovery(); + + // Wait a tiny bit to ensure timestamp would differ + const start = Date.now(); + while (Date.now() - start < 5) { + // Small wait + } + + const second = getCachedDiscovery(true); + + // If models are discovered, timestamps should differ + if (first.models.length > 0 || second.models.length > 0) { + assert.notStrictEqual( + first.timestamp, + second.timestamp, + "Force refresh should create new timestamp", + ); + } + }); +}); diff --git a/tests/evidence.test.ts b/tests/evidence.test.ts new file mode 100644 index 0000000..e363d9d --- /dev/null +++ b/tests/evidence.test.ts @@ -0,0 +1,276 @@ +/** + * evidence.test.ts — Tests for Phase 5: Lokale Crew-Evidence Collection + */ + +import { describe, it, beforeEach, afterEach } from "node:test"; +import assert from "node:assert"; +import fs from "node:fs"; +import path from "node:path"; +import { + recordEvidence, + loadEvidence, + aggregateEvidence, + loadAggregatedEvidence, + getEvidenceForTask, + getBestModelForTask, + recordFromNoMistakesOutcome, +} from "../src/evidence.ts"; + +describe("Evidence Collection", () => { + const evidenceDir = path.join(process.cwd(), "knowledge", "local"); + const evidencePath = path.join(evidenceDir, "evidence.json"); + const aggregatedPath = path.join(evidenceDir, "aggregated.json"); + + beforeEach(() => { + // Clear evidence before each test + if (fs.existsSync(evidencePath)) { + fs.unlinkSync(evidencePath); + } + if (fs.existsSync(aggregatedPath)) { + fs.unlinkSync(aggregatedPath); + } + }); + + afterEach(() => { + // Cleanup after tests + if (fs.existsSync(evidencePath)) { + fs.unlinkSync(evidencePath); + } + if (fs.existsSync(aggregatedPath)) { + fs.unlinkSync(aggregatedPath); + } + }); + + it("should record evidence", () => { + recordEvidence({ + taskId: "test-1", + taskClass: "standard_ship", + harness: "pi", + provider: "claude", + model: "sonnet", + effort: "medium", + startTime: Date.now() - 10000, + endTime: Date.now(), + durationMs: 10000, + success: true, + testsRun: 10, + testsPassed: 10, + ciResult: "pass", + reworkCount: 0, + }); + + const evidence = loadEvidence(); + assert.strictEqual(evidence.length, 1, "Should have one evidence entry"); + assert.strictEqual(evidence[0].taskId, "test-1", "Should match task ID"); + assert.strictEqual(evidence[0].source, "local", "Source should be local"); + assert.strictEqual(evidence[0].confidence, "measured", "Confidence should be measured"); + }); + + it("should aggregate evidence by task class and model", () => { + // Record multiple evidence entries + const baseTime = Date.now(); + + recordEvidence({ + taskId: "test-1", + taskClass: "standard_ship", + harness: "pi", + provider: "claude", + model: "sonnet", + effort: "medium", + startTime: baseTime - 10000, + endTime: baseTime, + durationMs: 10000, + success: true, + testsRun: 10, + testsPassed: 10, + ciResult: "pass", + reworkCount: 0, + }); + + recordEvidence({ + taskId: "test-2", + taskClass: "standard_ship", + harness: "pi", + provider: "claude", + model: "sonnet", + effort: "medium", + startTime: baseTime - 15000, + endTime: baseTime - 5000, + durationMs: 10000, + success: true, + testsRun: 10, + testsPassed: 9, + ciResult: "pass", + reworkCount: 1, + }); + + const aggregated = loadAggregatedEvidence(); + + assert.ok(aggregated.length > 0, "Should have aggregated data"); + + const entry = aggregated.find( + (e) => + e.taskClass === "standard_ship" && + e.provider === "claude" && + e.model === "sonnet", + ); + + assert.ok(entry, "Should find aggregated entry for standard_ship/claude/sonnet"); + assert.strictEqual(entry!.totalTasks, 2, "Should count 2 tasks"); + assert.strictEqual(entry!.successfulTasks, 2, "Should count 2 successes"); + assert.strictEqual(entry!.successRate, 1, "Success rate should be 100%"); + assert.strictEqual(entry!.avgDurationMs, 10000, "Average duration should be 10s"); + }); + + it("should get evidence for specific task/model", () => { + recordEvidence({ + taskId: "test-1", + taskClass: "trivial_fix", + harness: "claude", + provider: "claude", + model: "haiku", + effort: "low", + startTime: Date.now() - 5000, + endTime: Date.now(), + durationMs: 5000, + success: true, + reworkCount: 0, + }); + + const evidence = getEvidenceForTask("trivial_fix", "claude", "haiku"); + + assert.ok(evidence, "Should find evidence"); + assert.strictEqual(evidence!.taskClass, "trivial_fix", "Should match task class"); + assert.strictEqual(evidence!.provider, "claude", "Should match provider"); + assert.strictEqual(evidence!.model, "haiku", "Should match model"); + }); + + it("should identify best model for task class", () => { + const baseTime = Date.now(); + + // Claude performs better + recordEvidence({ + taskId: "test-1", + taskClass: "hard_multi_file", + harness: "pi", + provider: "claude", + model: "opus", + effort: "high", + startTime: baseTime - 30000, + endTime: baseTime, + durationMs: 30000, + success: true, + reworkCount: 0, + }); + + recordEvidence({ + taskId: "test-2", + taskClass: "hard_multi_file", + harness: "pi", + provider: "claude", + model: "opus", + effort: "high", + startTime: baseTime - 32000, + endTime: baseTime - 2000, + durationMs: 30000, + success: true, + reworkCount: 0, + }); + + // Codex has lower success rate + recordEvidence({ + taskId: "test-3", + taskClass: "hard_multi_file", + harness: "codex", + provider: "codex", + model: "gpt-5.5", + effort: "high", + startTime: baseTime - 40000, + endTime: baseTime - 10000, + durationMs: 30000, + success: false, + reworkCount: 2, + }); + + const best = getBestModelForTask("hard_multi_file"); + + assert.ok(best, "Should find best model"); + assert.strictEqual(best!.provider, "claude", "Claude should be best for hard_multi_file"); + assert.strictEqual(best!.model, "opus", "Opus should be best model"); + }); + + it("should record from no-mistakes outcome", () => { + const baseTime = Date.now(); + + recordFromNoMistakesOutcome({ + taskId: "nm-test-1", + taskClass: "standard_ship", + harness: "pi", + provider: "claude", + model: "sonnet", + effort: "medium", + startTime: baseTime - 20000, + endTime: baseTime, + success: true, + testsRun: 15, + testsPassed: 15, + ciResult: "pass", + reworkCount: 1, + }); + + const evidence = loadEvidence(); + + assert.strictEqual(evidence.length, 1, "Should record evidence"); + assert.strictEqual(evidence[0].taskId, "nm-test-1", "Should match task ID"); + assert.strictEqual(evidence[0].durationMs, 20000, "Should calculate duration"); + }); + + it("should handle missing test data gracefully", () => { + recordEvidence({ + taskId: "test-no-tests", + taskClass: "scout_audit", + harness: "pi", + provider: "claude", + model: "sonnet", + effort: "low", + startTime: Date.now() - 5000, + endTime: Date.now(), + durationMs: 5000, + success: true, + reworkCount: 0, + // No test data + }); + + const aggregated = loadAggregatedEvidence(); + const entry = aggregated.find((e) => e.taskClass === "scout_audit"); + + assert.ok(entry, "Should aggregate even without test data"); + // avgTestPassRate should handle missing data + assert.ok(typeof entry!.avgTestPassRate === "number", "Should have test pass rate"); + }); + + it("should preserve privacy - no code content", () => { + recordEvidence({ + taskId: "privacy-test", + taskClass: "standard_ship", + harness: "pi", + provider: "claude", + model: "sonnet", + effort: "medium", + startTime: Date.now() - 10000, + endTime: Date.now(), + durationMs: 10000, + success: true, + reworkCount: 0, + }); + + const evidence = loadEvidence(); + const entry = evidence[0]; + + // Ensure no code-related fields exist + assert.ok(!("code" in entry), "Should not contain code"); + assert.ok(!("prompt" in entry), "Should not contain prompt"); + assert.ok(!("diff" in entry), "Should not contain diff"); + assert.ok(!("files" in entry), "Should not contain files"); + }); +}); diff --git a/tests/knowledge-layers.test.ts b/tests/knowledge-layers.test.ts new file mode 100644 index 0000000..cc71a88 --- /dev/null +++ b/tests/knowledge-layers.test.ts @@ -0,0 +1,304 @@ +/** + * knowledge-layers.test.ts — Tests for Phase 2: Drei-Ebenen Knowledge + */ + +import { describe, it, beforeEach, afterEach } from "node:test"; +import assert from "node:assert"; +import fs from "node:fs"; +import path from "node:path"; +import { + saveManufacturerFact, + saveBenchmarkResult, + loadManufacturerFacts, + loadBenchmarkResults, + isValid, + filterValid, + resolveConflict, + getAvailability, + getCapability, + getQuality, +} from "../src/knowledge-layers.ts"; +import type { + ManufacturerFact, + BenchmarkResult, + CapabilityFact, +} from "../src/knowledge-layers.ts"; + +describe("Knowledge Layers", () => { + const manufacturersDir = path.join(process.cwd(), "knowledge", "manufacturers"); + const benchmarksDir = path.join(process.cwd(), "knowledge", "benchmarks"); + + beforeEach(() => { + // Clean up test files + if (fs.existsSync(manufacturersDir)) { + const files = fs.readdirSync(manufacturersDir).filter((f) => f.endsWith(".json")); + for (const file of files) { + fs.unlinkSync(path.join(manufacturersDir, file)); + } + } + if (fs.existsSync(benchmarksDir)) { + const files = fs.readdirSync(benchmarksDir).filter((f) => f.endsWith(".json")); + for (const file of files) { + fs.unlinkSync(path.join(benchmarksDir, file)); + } + } + }); + + afterEach(() => { + // Clean up after tests + if (fs.existsSync(manufacturersDir)) { + const files = fs.readdirSync(manufacturersDir).filter((f) => f.endsWith(".json")); + for (const file of files) { + fs.unlinkSync(path.join(manufacturersDir, file)); + } + } + if (fs.existsSync(benchmarksDir)) { + const files = fs.readdirSync(benchmarksDir).filter((f) => f.endsWith(".json")); + for (const file of files) { + fs.unlinkSync(path.join(benchmarksDir, file)); + } + } + }); + + it("should save and load manufacturer facts", () => { + const fact: ManufacturerFact = { + source: "official-docs", + confidence: "high", + timestamp: Date.now(), + provider: "claude", + fact: "Claude supports long context", + details: { contextWindow: 200000 }, + url: "https://docs.anthropic.com", + }; + + saveManufacturerFact(fact); + + const facts = loadManufacturerFacts(); + assert.ok(facts.length > 0, "Should load facts"); + + const loaded = facts.find((f) => f.provider === "claude"); + assert.ok(loaded, "Should find Claude fact"); + assert.strictEqual(loaded!.source, "official-docs", "Should have correct source"); + assert.strictEqual(loaded!.confidence, "high", "Should have high confidence"); + }); + + it("should save and load benchmark results", () => { + const benchmark: BenchmarkResult = { + source: "external-benchmark", + confidence: "medium", + timestamp: Date.now(), + benchmarkName: "HumanEval", + provider: "codex", + model: "gpt-5.5", + score: 95.5, + metric: "pass@1", + url: "https://example.com/benchmark", + }; + + saveBenchmarkResult(benchmark); + + const results = loadBenchmarkResults(); + assert.ok(results.length > 0, "Should load benchmarks"); + + const loaded = results.find( + (r) => r.benchmarkName === "HumanEval" && r.provider === "codex", + ); + assert.ok(loaded, "Should find HumanEval benchmark"); + assert.strictEqual(loaded!.score, 95.5, "Should have correct score"); + }); + + it("should validate knowledge entries by timestamp", () => { + const validEntry = { + source: "official-docs" as const, + confidence: "high" as const, + timestamp: Date.now(), + validUntil: Date.now() + 86400000, // 1 day from now + }; + + const expiredEntry = { + source: "official-docs" as const, + confidence: "high" as const, + timestamp: Date.now() - 86400000, // 1 day ago + validUntil: Date.now() - 3600000, // 1 hour ago + }; + + assert.ok(isValid(validEntry), "Should validate future expiry"); + assert.ok(!isValid(expiredEntry), "Should reject expired entry"); + }); + + it("should filter valid entries", () => { + const entries = [ + { + source: "official-docs" as const, + confidence: "high" as const, + timestamp: Date.now(), + validUntil: Date.now() + 86400000, + }, + { + source: "official-docs" as const, + confidence: "high" as const, + timestamp: Date.now() - 86400000, + validUntil: Date.now() - 3600000, + }, + ]; + + const valid = filterValid(entries); + assert.strictEqual(valid.length, 1, "Should filter out expired entry"); + }); + + it("should resolve conflicts by confidence priority", () => { + const entries = [ + { + source: "manufacturer-claim" as const, + confidence: "low" as const, + timestamp: Date.now(), + value: "claim", + }, + { + source: "external-benchmark" as const, + confidence: "medium" as const, + timestamp: Date.now(), + value: "benchmark", + }, + { + source: "local" as const, + confidence: "measured" as const, + timestamp: Date.now(), + value: "measured", + }, + ]; + + const best = resolveConflict(entries, () => true); + + assert.ok(best, "Should find best entry"); + assert.strictEqual(best!.confidence, "measured", "Should prefer measured"); + assert.strictEqual(best!.value, "measured", "Should select measured value"); + }); + + it("should prioritize newer entries when confidence equal", () => { + const now = Date.now(); + const entries = [ + { + source: "official-docs" as const, + confidence: "high" as const, + timestamp: now - 10000, + value: "old", + }, + { + source: "official-docs" as const, + confidence: "high" as const, + timestamp: now, + value: "new", + }, + ]; + + const best = resolveConflict(entries, () => true); + + assert.ok(best, "Should find best entry"); + assert.strictEqual(best!.value, "new", "Should prefer newer entry"); + }); + + it("should get availability with correct priority", () => { + const entries = [ + { + source: "manufacturer-claim" as const, + confidence: "low" as const, + timestamp: Date.now(), + provider: "test", + model: "test-model", + }, + { + source: "discovery" as const, + confidence: "high" as const, + timestamp: Date.now(), + provider: "test", + model: "test-model", + }, + ]; + + const availability = getAvailability("test", "test-model", entries); + + assert.ok(availability.available, "Should be available"); + assert.strictEqual( + availability.source, + "discovery", + "Should prefer discovery over manufacturer claim", + ); + }); + + it("should get capability with correct priority", () => { + const capabilities: CapabilityFact[] = [ + { + source: "manufacturer-claim", + confidence: "low", + timestamp: Date.now(), + provider: "test", + model: "test-model", + capability: "long-context", + supported: true, + }, + { + source: "official-docs", + confidence: "high", + timestamp: Date.now(), + provider: "test", + model: "test-model", + capability: "long-context", + supported: true, + }, + ]; + + const capability = getCapability("test", "test-model", "long-context", capabilities); + + assert.ok(capability.supported, "Should be supported"); + assert.strictEqual( + capability.source, + "official-docs", + "Should prefer official docs over manufacturer claim", + ); + }); + + it("should get quality with correct priority", () => { + const entries = [ + { + source: "manufacturer-claim" as const, + confidence: "low" as const, + timestamp: Date.now(), + provider: "test", + model: "test-model", + score: 90, + }, + { + source: "local" as const, + confidence: "measured" as const, + timestamp: Date.now(), + provider: "test", + model: "test-model", + score: 85, + }, + ]; + + const quality = getQuality("test", "test-model", entries); + + assert.ok(quality.score, "Should have score"); + assert.strictEqual(quality.score, 85, "Should prefer local evidence over claim"); + assert.strictEqual(quality.source, "local", "Should use local source"); + assert.strictEqual(quality.confidence, "measured", "Should have measured confidence"); + }); + + it("should handle missing knowledge gracefully", () => { + const entries: any[] = []; + + const availability = getAvailability("nonexistent", "model", entries); + assert.strictEqual(availability.available, false, "Should not be available"); + assert.strictEqual(availability.confidence, "uncertain", "Should be uncertain"); + + const capability = getCapability("nonexistent", "model", "test", entries); + assert.strictEqual(capability.supported, false, "Should not be supported"); + assert.strictEqual(capability.confidence, "uncertain", "Should be uncertain"); + + const quality = getQuality("nonexistent", "model", entries); + assert.strictEqual(quality.score, undefined, "Should have no score"); + assert.strictEqual(quality.confidence, "uncertain", "Should be uncertain"); + }); +}); diff --git a/tests/update.test.ts b/tests/update.test.ts new file mode 100644 index 0000000..1a2f8e8 --- /dev/null +++ b/tests/update.test.ts @@ -0,0 +1,159 @@ +/** + * update.test.ts — Tests for Phase 3: Selbstaktualisierung + */ + +import { describe, it } from "node:test"; +import assert from "node:assert"; +import { + checkCompatibility, + getFirstmateVersion, + needsUpdate, +} from "../src/update.ts"; + +describe("Update and Compatibility", () => { + it("should check compatibility", () => { + const result = checkCompatibility(); + + assert.ok(result, "Should return compatibility result"); + assert.ok(typeof result.compatible === "boolean", "Should have compatible flag"); + assert.ok(Array.isArray(result.checks), "Should have checks array"); + assert.ok(Array.isArray(result.errors), "Should have errors array"); + assert.ok(Array.isArray(result.warnings), "Should have warnings array"); + }); + + it("should report check status", () => { + const result = checkCompatibility(); + + for (const check of result.checks) { + assert.ok(check.check, "Check should have name"); + assert.ok(["ok", "warning", "error"].includes(check.status), "Check should have valid status"); + assert.ok(check.message, "Check should have message"); + } + }); + + it("should check dispatch schema", () => { + const result = checkCompatibility(); + const dispatchCheck = result.checks.find((c) => c.check === "dispatch-schema"); + + assert.ok(dispatchCheck, "Should have dispatch schema check"); + // Should be ok or warning (might not have FM_HOME in test env) + assert.ok( + ["ok", "warning"].includes(dispatchCheck.status), + "Dispatch schema should not error in normal conditions", + ); + }); + + it("should check harnesses", () => { + const result = checkCompatibility(); + const harnessCheck = result.checks.find((c) => c.check === "harnesses"); + + assert.ok(harnessCheck, "Should have harness check"); + assert.ok( + ["ok", "warning"].includes(harnessCheck.status), + "Harness check should not error", + ); + }); + + it("should check spawn flags", () => { + const result = checkCompatibility(); + const flagsCheck = result.checks.find((c) => c.check === "spawn-flags"); + + assert.ok(flagsCheck, "Should have spawn flags check"); + // Might be warning if FM_HOME not set + assert.ok( + ["ok", "warning"].includes(flagsCheck.status), + "Spawn flags check should be ok or warning", + ); + }); + + it("should check effort values", () => { + const result = checkCompatibility(); + const effortCheck = result.checks.find((c) => c.check === "effort-values"); + + assert.ok(effortCheck, "Should have effort values check"); + assert.strictEqual(effortCheck.status, "ok", "Effort values should be ok"); + }); + + it("should check quota-axi availability", () => { + const result = checkCompatibility(); + const quotaCheck = result.checks.find((c) => c.check === "quota-axi"); + + assert.ok(quotaCheck, "Should have quota-axi check"); + // Can be ok or warning depending on environment + assert.ok( + ["ok", "warning"].includes(quotaCheck.status), + "Quota-axi check should be ok or warning", + ); + }); + + it("should get Firstmate version if available", () => { + const version = getFirstmateVersion(); + + // Might be undefined in test environment without FM_HOME + if (version) { + assert.ok(typeof version === "string", "Version should be string"); + assert.ok(version.length > 0, "Version should not be empty"); + } + }); + + it("should detect when update is needed", () => { + const needsUpdateResult = needsUpdate("0.0.1"); + + // Result depends on whether FM_HOME is set and has version + assert.ok( + typeof needsUpdateResult === "boolean", + "Should return boolean", + ); + }); + + it("should not need update when versions match", () => { + const currentVersion = getFirstmateVersion(); + + if (currentVersion) { + const needsUpdateResult = needsUpdate(currentVersion); + assert.strictEqual( + needsUpdateResult, + false, + "Should not need update when versions match", + ); + } + }); + + it("should collect errors and warnings separately", () => { + const result = checkCompatibility(); + + // Errors should only come from error-status checks + const errorChecks = result.checks.filter((c) => c.status === "error"); + assert.strictEqual( + result.errors.length, + errorChecks.length, + "Errors array should match error checks", + ); + + // Warnings should only come from warning-status checks + const warningChecks = result.checks.filter((c) => c.status === "warning"); + assert.strictEqual( + result.warnings.length, + warningChecks.length, + "Warnings array should match warning checks", + ); + }); + + it("should be compatible if no errors", () => { + const result = checkCompatibility(); + + if (result.errors.length === 0) { + assert.strictEqual( + result.compatible, + true, + "Should be compatible when no errors", + ); + } else { + assert.strictEqual( + result.compatible, + false, + "Should not be compatible when errors exist", + ); + } + }); +});