diff --git a/packages/std/ops/README.md b/packages/std/ops/README.md index e0aa9816..edf304b6 100644 --- a/packages/std/ops/README.md +++ b/packages/std/ops/README.md @@ -19,7 +19,7 @@ come back as `unresolved-intent` instead of terminal prompts. | `preflight.prose.md` | `prose preflight ` | Check that dependencies are installed and environment variables are set | | `wire.prose.md` | `prose run std/ops/wire` | Run Forme wiring to produce an execution manifest | | `status.prose.md` | `prose status` | Show recent runs with system name, duration, and pass/fail status | -| `prose-author.prose.md` | `prose write [--out ] [--apply] [--run] [request...]` | Interactive-by-default authoring of a validated OpenProse package from rough English or pseudo-Prose; `--out --apply` may write it, and host adapters that support `--run` expand it to write/apply followed by ordinary `prose run ` | +| `prose-author.prose.md` | `prose write [--out ] [--apply] [--run] [--test-iterations <0-3>] [request...]` | Interactive-by-default authoring of a validated OpenProse package from rough English or pseudo-Prose; `--out --apply` may write it, CLI host adapters may test generated `kind: test` files after apply, and `--run` expands to ordinary `prose run ` | | `diagnose.prose.md` | `prose run std/ops/diagnose` | Diagnose why a run failed -- root cause analysis with fix recommendations | | `profiler.prose.md` | `prose run std/ops/profiler` | Profile a run for cost, tokens, and time using actual API session data | diff --git a/skills/open-prose/SKILL.md b/skills/open-prose/SKILL.md index b2aa751a..733b599d 100644 --- a/skills/open-prose/SKILL.md +++ b/skills/open-prose/SKILL.md @@ -128,7 +128,7 @@ executing the system. The shell executable is the agent runner, e.g. | `prose run runtime/judge-responsibility.prose.md` | Resolve from the OpenProse skill root; judge one responsibility from activation context | | `prose run //[/path]` | Resolve installed dependency service or system, detect format, then route as above | | `prose run std/...` / `co/...` | Expand OpenProse package shorthand, resolve installed dependency service or system, then route as above | -| `prose write [--out ] [--apply] [--run] [request...]` | Interactive-by-default authoring: load `contract-markdown.md`, `guidance/tenets.md`, and `guidance/authoring.md`; run `std/ops/prose-author`; scan the local landscape read-only, decide shape/root/path, load shape-specific guidance, ask a small number of targeted `ask_user` questions when the host can support them, then return a fully validated source package. If the caller or host marks the run non-interactive, return `unresolved-intent` with the missing decisions instead of guessing. Apply generated files only when the caller explicitly passes apply permission, normally through `--out --apply`; `--run` is a host-adapter macro that implies apply permission and expands to an ordinary `prose run ` only after authoring succeeds | +| `prose write [--out ] [--apply] [--run] [--test-iterations <0-3>] [request...]` | Interactive-by-default authoring: load `contract-markdown.md`, `guidance/tenets.md`, and `guidance/authoring.md`; run `std/ops/prose-author`; scan the local landscape read-only, decide shape/root/path, load shape-specific guidance, ask a small number of targeted `ask_user` questions when the host can support them, then return a fully validated source package. If the caller or host marks the run non-interactive, return `unresolved-intent` with the missing decisions instead of guessing. Apply generated files only when the caller explicitly passes apply permission, normally through `--out --apply`; `--run` is a host-adapter macro that implies apply permission and expands to an ordinary `prose run ` only after authoring succeeds. `--test-iterations` is a CLI host-adapter macro with default `1`; `0` disables generated-test execution, and `3` is the maximum bounded repair loop | | `prose lint ` | Validate Contract Markdown structure, headers, frontmatter, contracts, shapes, and wiring | | `prose preflight ` | Check dependencies and `### Environment` declarations without executing | | `prose test ` | Load `contract-markdown.md`, `state/README.md` plus the selected backend, and `prose.md`; run `kind: test` file(s) | @@ -149,6 +149,18 @@ authoring succeeds. A host that cannot perform this macro must reject `prose write --run` before authoring and must not pass the macro into `prose-author`. +`prose write --test-iterations` is also a host-adapter macro for apply-enabled +CLI writes. The CLI host adapter defaults to one generated-test attempt: it +runs ordinary `prose test` for generated `kind: test` files after authoring +succeeds, repairs by invoking a new apply-enabled `prose-author` pass only +between failed test attempts, and stops at the requested bound. The macro is +off when set to `0`, and the maximum is `3`. Plain in-session `prose write` +still routes to `prose-author`; a non-CLI host that cannot perform this macro +must reject an explicit `--test-iterations` option before authoring, must not +pass the test-iteration macro into `prose-author`, and must not perform +optional giving-back, memory, or mycelium note side effects while executing a +forwarded/non-interactive write loop. + There is one skill: `open-prose`. Do not look for separate `prose-run`, `prose-lint`, `prose-compile`, or `prose-boot` skills. diff --git a/skills/open-prose/help.md b/skills/open-prose/help.md index 34f484bd..efc8b392 100644 --- a/skills/open-prose/help.md +++ b/skills/open-prose/help.md @@ -48,7 +48,7 @@ Options: | `prose compile [path] [--out ]` | Compile source into `/dist/manifest.next.json` | | `prose serve` | Serve the active IR as local cron and HTTP trigger adapters | | `prose run ` | Run a service or system | -| `prose write [--out ] [--apply] [--run] [request...]` | Interactive-by-default authoring from rough English/pseudo-Prose into a validated source package; `--out --apply` writes it after lint passes, and host adapters that support `--run` expand it to write/apply followed by ordinary `prose run ` | +| `prose write [--out ] [--apply] [--run] [--test-iterations <0-3>] [request...]` | Interactive-by-default authoring from rough English/pseudo-Prose into a validated source package; `--out --apply` writes it after lint passes, supported host adapters run generated tests after apply, and `--run` expands to ordinary `prose run ` | | `prose lint ` | Validate structure, schema, and contracts | | `prose preflight ` | Check dependencies and environment | | `prose test ` | Run tests with assertions | diff --git a/skills/open-prose/prose.md b/skills/open-prose/prose.md index 1bc5b304..cd777830 100644 --- a/skills/open-prose/prose.md +++ b/skills/open-prose/prose.md @@ -45,7 +45,7 @@ codex exec "prose run system.prose.md" | `prose run /` | Reserved for the OpenProse registry (future home at `p.prose.md`) | | `prose run ...@` | Pin to a SHA or tag; require that version in `/deps/` | | `prose run ... --offline` | Require disk-only resolution; error if not in `/deps/` | -| `prose write [--out ] [--apply] [--run] [request...]` | Interactive-by-default authoring through `std/ops/prose-author`, asking targeted shape/root questions when supported and returning a validated source package; `--out --apply` writes it when permitted, and host adapters that support `--run` expand it to write/apply followed by ordinary `prose run ` | +| `prose write [--out ] [--apply] [--run] [--test-iterations <0-3>] [request...]` | Interactive-by-default authoring through `std/ops/prose-author`, asking targeted shape/root questions when supported and returning a validated source package; `--out --apply` writes it when permitted, host adapters that support `--test-iterations` run generated tests after apply, and host adapters that support `--run` expand it to write/apply followed by ordinary `prose run ` | | `prose lint ` | Validate structure, schema, shapes, and contracts | | `prose preflight ` | Check dependencies, declared tools, and environment variables | | `prose test ` | Run test(s) and report results | diff --git a/tools/cli/README.md b/tools/cli/README.md index 728e05ea..cb68d4ba 100644 --- a/tools/cli/README.md +++ b/tools/cli/README.md @@ -77,6 +77,7 @@ prose run std/evals/inspector prose run std/evals/prose-contributor -- subjects: 20260406-201439-1a3369 prose write "draft a release readiness responsibility" prose write --out src/release-readiness --run "draft a release readiness responsibility" +prose write --out src/release-readiness --apply --test-iterations=3 "draft a release readiness responsibility" cat brief.txt | prose write --harness codex-sdk prose run std/evals/inspector --harness codex-sdk prose run co/systems/company-repo-checker --harness claude-sdk @@ -169,6 +170,18 @@ then starts a separate ordinary `prose run` for the generated root file only after authoring succeeds. Directory targets run `/index.prose.md`; file targets must end in `.prose.md`. +Generated-test execution is also a CLI host-adapter macro. After apply-enabled +authoring succeeds, the CLI discovers generated `kind: test` files under +`--out` and runs ordinary `prose test` for each one. `--test-iterations=1` is +the default for `--apply` and `--run`, `--test-iterations=0` disables the test +loop, and `--test-iterations=3` allows up to three test attempts with repair +authoring passes between failures. The test loop never commits generated files and does +not ask the authoring run to perform optional giving-back, memory, or mycelium +note side effects. Repair prompts include captured failing test output capped +at 12 KB; the side-effect ban and target-path repair instruction are prepended +before that output. If the loop exhausts its iteration bound, the CLI returns +the final failing `prose test` exit code rather than a distinct sentinel. + `--out` is a validated, root-relative target contract passed to the authoring harness. The CLI rejects absolute paths, parent traversal, and file targets that do not end in `.prose.md`; the actual write is still performed by the @@ -184,6 +197,7 @@ prose run src/systems/reviewer.prose.md prose write "draft a release readiness responsibility" prose write --out src/reviewer --apply "draft a reviewer system" prose write --out src/reviewer --run "draft a reviewer system" +prose write --out src/reviewer --apply --test-iterations=0 "draft a reviewer system" prose write < brief.txt prose run co/systems/company-repo-checker --harness claude-sdk prose upgrade diff --git a/tools/cli/src/commands/base.ts b/tools/cli/src/commands/base.ts index a4144f90..aaa5019a 100644 --- a/tools/cli/src/commands/base.ts +++ b/tools/cli/src/commands/base.ts @@ -1,5 +1,7 @@ +import { existsSync, lstatSync, readdirSync, readFileSync } from "node:fs"; +import { join } from "node:path"; import { Command } from "@oclif/core"; -import type { CommandName } from "../prose/index.js"; +import type { CommandName, WriteCommandOptions } from "../prose/index.js"; import { canonicalPrompt, CommandModelError, parseWriteCommand, resolveWriteRunTarget, usageFor } from "../prose/index.js"; import { recordForwardedFulfillmentArtifact } from "../prose/fulfillment-artifact.js"; import { createHarness, type HarnessName } from "../harnesses/index.js"; @@ -94,8 +96,9 @@ function isOclifExit(error: unknown): boolean { export async function runForwardedProseCommand(options: ForwardRunOptions): Promise { const { harness, args } = splitHarnessArgs(options.argv, options.env, options.command); const promptArgs = await hydrateForwardedArgs(options.command, args, options.stdin); + const writeOptions = options.command === "write" ? parseWriteCommand(promptArgs) : undefined; const prompt = canonicalPrompt(options.command, promptArgs); - const writeRequiresFilesystem = options.command === "write" ? parseWriteCommand(promptArgs).apply : false; + const writeRequiresFilesystem = writeOptions?.apply ?? false; const writeRunTarget = options.command === "write" ? resolveWriteRunTarget(promptArgs) : undefined; if (shouldRunSkillPreflight(options)) { await runSkillPreflight(harness, options); @@ -109,6 +112,7 @@ export async function runForwardedProseCommand(options: ForwardRunOptions): Prom harness, requiresFilesystemWrites: writeRequiresFilesystem, }); + const writeTestBaseline = snapshotWriteTestBaseline(options.cwd, writeOptions); const exitCode = await selectedHarness.run(prompt, harnessRunOptions); await recordForwardedFulfillmentArtifact({ command: options.command, @@ -120,13 +124,31 @@ export async function runForwardedProseCommand(options: ForwardRunOptions): Prom prompt, }); - if (exitCode !== 0 || writeRunTarget === undefined) { + if (exitCode !== 0) { return exitCode; } if (options.signal?.aborted) { return 143; } + const testLoopExitCode = await runWriteTestLoop({ + forwardOptions: options, + harness, + harnessRunOptions, + selectedHarness, + writeTestBaseline, + writeOptions, + }); + if (testLoopExitCode !== 0) { + return testLoopExitCode; + } + if (options.signal?.aborted) { + return 143; + } + if (writeRunTarget === undefined) { + return exitCode; + } + const runPrompt = canonicalPrompt("run", [writeRunTarget]); const runExitCode = await selectedHarness.run(runPrompt, harnessRunOptions); await recordForwardedFulfillmentArtifact({ @@ -141,6 +163,292 @@ export async function runForwardedProseCommand(options: ForwardRunOptions): Prom return runExitCode; } +interface WriteTestLoopOptions { + forwardOptions: ForwardRunOptions; + harness: string; + harnessRunOptions: HarnessRunOptions; + selectedHarness: Harness; + writeTestBaseline: ReadonlyMap | undefined; + writeOptions: WriteCommandOptions | undefined; +} + +interface WriteTestResult { + exitCode: number; + output: string; + prompt: string; +} + +async function runWriteTestLoop(options: WriteTestLoopOptions): Promise { + const write = options.writeOptions; + if (write === undefined || !write.apply || write.out === undefined || write.testIterations === 0) { + return 0; + } + + let requiredTestTargets: readonly string[] | undefined; + for (let attempt = 1; attempt <= write.testIterations; attempt += 1) { + if (options.forwardOptions.signal?.aborted) { + return 143; + } + + const discoveredTargets = discoverGeneratedWriteTestTargets(options.forwardOptions.cwd, write.out); + const changedTargets = filterChangedWriteTestTargets( + options.forwardOptions.cwd, + discoveredTargets, + options.writeTestBaseline, + ); + requiredTestTargets ??= changedTargets; + const missingRequiredTargets = requiredTestTargets.filter((target) => !discoveredTargets.includes(target)); + if (missingRequiredTargets.length > 0) { + options.forwardOptions.stderr.write( + `Generated test target disappeared after repair: ${missingRequiredTargets.join(", ")}\n`, + ); + return 1; + } + + const testTargets = mergeTestTargets(requiredTestTargets, changedTargets); + if (testTargets.length === 0) { + return 0; + } + + const testResult = await runGeneratedWriteTests(options, testTargets); + if (testResult.exitCode === 0) { + return 0; + } + if (attempt === write.testIterations) { + return testResult.exitCode; + } + if (options.forwardOptions.signal?.aborted) { + return 143; + } + + const repairRequest = buildWriteTestRepairRequest(write, testResult); + const repairArgs = ["--out", write.out, "--apply", repairRequest]; + const repairPrompt = canonicalPrompt("write", repairArgs); + const repairExitCode = await options.selectedHarness.run(repairPrompt, options.harnessRunOptions); + await recordForwardedFulfillmentArtifact({ + command: "write", + argv: repairArgs, + cwd: options.forwardOptions.cwd, + env: options.forwardOptions.env, + exitCode: repairExitCode, + harness: options.harness, + prompt: repairPrompt, + }); + if (repairExitCode !== 0) { + return repairExitCode; + } + } + + return 0; +} + +function mergeTestTargets(required: readonly string[], changed: readonly string[]): string[] { + return Array.from(new Set([...required, ...changed])).sort(); +} + +async function runGeneratedWriteTests(options: WriteTestLoopOptions, testTargets: readonly string[]): Promise { + for (const testTarget of testTargets) { + const testPrompt = canonicalPrompt("test", [testTarget]); + const capture = createCapturedHarnessStreams(options.harnessRunOptions.stdout, options.harnessRunOptions.stderr); + const testExitCode = await options.selectedHarness.run(testPrompt, { + ...options.harnessRunOptions, + stdout: capture.stdout, + stderr: capture.stderr, + }); + await recordForwardedFulfillmentArtifact({ + command: "test", + argv: [testTarget], + cwd: options.forwardOptions.cwd, + env: options.forwardOptions.env, + exitCode: testExitCode, + harness: options.harness, + prompt: testPrompt, + }); + if (testExitCode !== 0) { + return { + exitCode: testExitCode, + output: capture.output, + prompt: testPrompt, + }; + } + } + + return { + exitCode: 0, + output: "", + prompt: "", + }; +} + +function buildWriteTestRepairRequest(write: WriteCommandOptions, testResult: WriteTestResult): string { + const parts = [ + `Repair the generated OpenProse source under \`${write.out ?? ""}\` after a host-managed test iteration failed.`, + `Original request: ${write.request}`, + `Failing test command: ${testResult.prompt}`, + `Failing test exit code: ${testResult.exitCode}`, + "Read the generated files under target_path and apply only source repairs under that same target.", + "Keep forwarded/non-interactive write boundaries: do not run tests yourself, do not run the generated root, and do not perform optional giving-back, memory, or mycelium note side effects.", + ]; + const output = testResult.output.trim(); + if (output !== "") { + parts.push(`Captured test output:\n${output}`); + } + return parts.join("\n\n"); +} + +function discoverGeneratedWriteTestTargets(cwd: string, targetPath: string): string[] { + const root = join(cwd, ...targetPath.split("/")); + if (!existsSync(root)) { + return []; + } + + const targets: string[] = []; + visitGeneratedWriteTarget(root, targetPath === "." ? "" : targetPath, targets); + return targets.sort(); +} + +function snapshotWriteTestBaseline( + cwd: string, + write: WriteCommandOptions | undefined, +): ReadonlyMap | undefined { + if (write === undefined || !write.apply || write.out === undefined || write.testIterations === 0) { + return undefined; + } + + return new Map( + discoverGeneratedWriteTestTargets(cwd, write.out) + .map((target): [string, string] | undefined => { + const source = readRootRelativeFileIfExists(cwd, target); + return source === undefined ? undefined : [target, source]; + }) + .filter((entry): entry is [string, string] => entry !== undefined), + ); +} + +function filterChangedWriteTestTargets( + cwd: string, + targets: readonly string[], + baseline: ReadonlyMap | undefined, +): string[] { + if (baseline === undefined) { + return [...targets]; + } + + return targets.filter((target) => { + const source = readRootRelativeFileIfExists(cwd, target); + return source !== undefined && baseline.get(target) !== source; + }); +} + +function visitGeneratedWriteTarget(fsPath: string, relativePath: string, targets: string[]): void { + let stat; + try { + stat = lstatSync(fsPath); + } catch { + return; + } + if (stat.isSymbolicLink()) { + return; + } + if (stat.isDirectory()) { + let entries: string[]; + try { + entries = readdirSync(fsPath).sort(); + } catch { + return; + } + for (const entry of entries) { + visitGeneratedWriteTarget(join(fsPath, entry), relativePath === "" ? entry : `${relativePath}/${entry}`, targets); + } + return; + } + if (!stat.isFile() || !relativePath.endsWith(".prose.md")) { + return; + } + if (relativePath.endsWith(".test.prose.md") || proseFileHasTestKind(fsPath)) { + targets.push(relativePath); + } +} + +function proseFileHasTestKind(fsPath: string): boolean { + const source = readFileIfExists(fsPath); + return source !== undefined && parseFlatFrontmatter(source).get("kind") === "test"; +} + +function parseFlatFrontmatter(source: string): Map { + const frontmatter = source.match(/^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/); + if (frontmatter?.[1] === undefined) { + return new Map(); + } + + const parsed = new Map(); + for (const line of frontmatter[1].split(/\r?\n/)) { + const match = line.match(/^([A-Za-z][A-Za-z0-9_-]*)\s*:\s*(.*?)\s*$/); + if (match?.[1] !== undefined && match[2] !== undefined) { + parsed.set(match[1], stripYamlScalarQuotes(match[2].trim())); + } + } + return parsed; +} + +function stripYamlScalarQuotes(value: string): string { + if (value.length >= 2 && value.startsWith('"') && value.endsWith('"')) { + return value.slice(1, -1); + } + if (value.length >= 2 && value.startsWith("'") && value.endsWith("'")) { + return value.slice(1, -1); + } + return value; +} + +function readRootRelativeFileIfExists(cwd: string, target: string): string | undefined { + return readFileIfExists(join(cwd, ...target.split("/"))); +} + +function readFileIfExists(fsPath: string): string | undefined { + try { + return readFileSync(fsPath, "utf8"); + } catch { + return undefined; + } +} + +function createCapturedHarnessStreams(stdout: WritableStreamLike, stderr: WritableStreamLike) { + const maxOutputLength = 12_000; + let output = ""; + let truncated = false; + + function capture(label: "stdout" | "stderr", chunk: string): void { + const formatted = `[${label}] ${chunk}`; + const remaining = maxOutputLength - output.length; + if (remaining > 0) { + output += formatted.slice(0, remaining); + } + if (formatted.length > remaining && !truncated) { + output += "\n[truncated]\n"; + truncated = true; + } + } + + return { + stdout: { + write(chunk: string) { + capture("stdout", chunk); + return stdout.write(chunk); + }, + }, + stderr: { + write(chunk: string) { + capture("stderr", chunk); + return stderr.write(chunk); + }, + }, + get output() { + return output; + }, + }; +} + function buildHarnessRunOptions( options: ForwardRunOptions, skillBootstrap: OpenProseSkillBootstrap | undefined, diff --git a/tools/cli/src/commands/index.ts b/tools/cli/src/commands/index.ts index 1fa2420d..9dcf01f8 100644 --- a/tools/cli/src/commands/index.ts +++ b/tools/cli/src/commands/index.ts @@ -66,10 +66,11 @@ const forwardCommandDefinitions = { examples: [ '<%= config.bin %> write "draft a release readiness responsibility" --harness codex-sdk', '<%= config.bin %> write --out src/release-readiness --run "draft a release readiness responsibility"', + '<%= config.bin %> write --out src/release-readiness --apply --test-iterations=3 "draft a release readiness responsibility"', "cat brief.txt | <%= config.bin %> write --harness codex-sdk", ], summary: "Write linted OpenProse source from rough intent.", - usage: "write [--out ] [--apply] [--run] [request...] [--harness ]", + usage: "write [--out ] [--apply] [--run] [--test-iterations <0-3>] [request...] [--harness ]", }, } satisfies Record; diff --git a/tools/cli/src/prose/command-model.ts b/tools/cli/src/prose/command-model.ts index 2e9bdb6b..a97ba370 100644 --- a/tools/cli/src/prose/command-model.ts +++ b/tools/cli/src/prose/command-model.ts @@ -42,7 +42,7 @@ export const supportedCommands = [ const usageByCommand: Record = { compile: "prose compile [path] [--out ]", run: "prose run [inputs...]", - write: "prose write [--out ] [--apply] [--run] [request...]", + write: "prose write [--out ] [--apply] [--run] [--test-iterations <0-3>] [request...]", lint: "prose lint ", preflight: "prose preflight ", test: "prose test ", @@ -90,6 +90,7 @@ export interface WriteCommandOptions { out?: string; request: string; run: boolean; + testIterations: number; } export function parseWriteCommand(args: readonly string[]): WriteCommandOptions { @@ -98,6 +99,8 @@ export function parseWriteCommand(args: readonly string[]): WriteCommandOptions let out: string | undefined; let apply = false; let run = false; + let testIterations = 1; + let sawTestIterations = false; for (let index = 0; index < args.length; index += 1) { const arg = args[index]; @@ -129,6 +132,29 @@ export function parseWriteCommand(args: readonly string[]): WriteCommandOptions continue; } + if (arg === "--test-iterations") { + if (sawTestIterations) { + fail("write", "Duplicate option for 'prose write'."); + } + const value = args[index + 1]; + if (value === undefined || value === "" || value.startsWith("-")) { + fail("write", "Missing value for --test-iterations."); + } + testIterations = normalizeWriteTestIterations(value); + sawTestIterations = true; + index += 1; + continue; + } + + if (arg.startsWith("--test-iterations=")) { + if (sawTestIterations) { + fail("write", "Duplicate option for 'prose write'."); + } + testIterations = normalizeWriteTestIterations(arg.slice("--test-iterations=".length)); + sawTestIterations = true; + continue; + } + if (arg === "--out") { if (out !== undefined) { fail("write", "Duplicate option for 'prose write'."); @@ -167,6 +193,7 @@ export function parseWriteCommand(args: readonly string[]): WriteCommandOptions ...(out === undefined ? {} : { out }), request, run, + testIterations, }; } @@ -331,6 +358,20 @@ function normalizeWriteTargetPath(value: string): string { return normalized === "." ? "." : normalized; } +function normalizeWriteTestIterations(value: string): number { + if (value.trim() === "") { + fail("write", "Missing value for --test-iterations."); + } + if (!/^\d+$/.test(value)) { + fail("write", "--test-iterations must be an integer between 0 and 3."); + } + const parsed = Number(value); + if (!Number.isSafeInteger(parsed) || parsed < 0 || parsed > 3) { + fail("write", "--test-iterations must be an integer between 0 and 3."); + } + return parsed; +} + function rootFileForWriteTarget(out: string): string { if (out.endsWith(".prose.md")) { return out; diff --git a/tools/cli/tests/cli/cli.test.ts b/tools/cli/tests/cli/cli.test.ts index e0c7ead6..689cc1d7 100644 --- a/tools/cli/tests/cli/cli.test.ts +++ b/tools/cli/tests/cli/cli.test.ts @@ -517,6 +517,299 @@ kind: system ]); }); + it("runs generated test files before the write-run follow-up", async () => { + const cwd = mkdtempSync(join(tmpdir(), "prose-write-test-run-")); + const io = memoryStreams(); + const seen: string[] = []; + const targetDir = join(cwd, "src", "release-readiness"); + + try { + const exitCode = await runForwardedProseCommand({ + command: "write", + argv: ["--out", "src/release-readiness", "--run", "draft release readiness", "--harness", "mock"], + cwd, + env: {}, + stdout: io.streams.stdout, + stderr: io.streams.stderr, + harnessFactory: () => ({ + name: "mock", + async run(prompt) { + seen.push(prompt); + if (prompt.startsWith("prose write ")) { + mkdirSync(targetDir, { recursive: true }); + writeFileSync(join(targetDir, "index.prose.md"), "---\nname: release-readiness\nkind: system\n---\n"); + writeFileSync( + join(targetDir, "release-readiness.test.prose.md"), + "---\nname: release-readiness-test\nkind: test\nsubject: release-readiness\n---\n", + ); + } + return 0; + }, + }), + }); + + expect(exitCode).toBe(0); + expect(seen).toEqual([ + "prose write output_mode: source-package-and-files apply: true target_path: src/release-readiness post_apply_action: host-will-run-root run_state: filesystem terminal_summary: required interactive: false request: 'draft release readiness'", + "prose test src/release-readiness/release-readiness.test.prose.md", + "prose run src/release-readiness/index.prose.md", + ]); + } finally { + rmSync(cwd, { recursive: true, force: true }); + } + }); + + it("discovers generated tests from quoted kind frontmatter without body false positives", async () => { + const cwd = mkdtempSync(join(tmpdir(), "prose-write-test-frontmatter-")); + const io = memoryStreams(); + const seen: string[] = []; + const targetDir = join(cwd, "src", "release-readiness"); + + try { + const exitCode = await runForwardedProseCommand({ + command: "write", + argv: ["--out", "src/release-readiness", "--apply", "draft release readiness", "--harness", "mock"], + cwd, + env: {}, + stdout: io.streams.stdout, + stderr: io.streams.stderr, + harnessFactory: () => ({ + name: "mock", + async run(prompt) { + seen.push(prompt); + if (prompt.startsWith("prose write ")) { + mkdirSync(targetDir, { recursive: true }); + writeFileSync(join(targetDir, "index.prose.md"), "---\nname: release-readiness\nkind: system\n---\n"); + writeFileSync( + join(targetDir, "quoted-kind.prose.md"), + '---\nname: quoted-kind\nkind: "test"\nsubject: release-readiness\n---\n', + ); + writeFileSync( + join(targetDir, "notes.prose.md"), + "---\nname: release-notes\nkind: system\n---\n\nThis body mentions kind: test as prose, not frontmatter.\n", + ); + } + return 0; + }, + }), + }); + + expect(exitCode).toBe(0); + expect(seen).toEqual([ + "prose write output_mode: source-package-and-files apply: true target_path: src/release-readiness post_apply_action: none run_state: filesystem terminal_summary: required interactive: false request: 'draft release readiness'", + "prose test src/release-readiness/quoted-kind.prose.md", + ]); + } finally { + rmSync(cwd, { recursive: true, force: true }); + } + }); + + it("skips generated test files when write test iterations are disabled", async () => { + const cwd = mkdtempSync(join(tmpdir(), "prose-write-test-off-")); + const io = memoryStreams(); + const seen: string[] = []; + const targetDir = join(cwd, "src", "release-readiness"); + + try { + const exitCode = await runForwardedProseCommand({ + command: "write", + argv: [ + "--out", + "src/release-readiness", + "--run", + "--test-iterations=0", + "draft release readiness", + "--harness", + "mock", + ], + cwd, + env: {}, + stdout: io.streams.stdout, + stderr: io.streams.stderr, + harnessFactory: () => ({ + name: "mock", + async run(prompt) { + seen.push(prompt); + if (prompt.startsWith("prose write ")) { + mkdirSync(targetDir, { recursive: true }); + writeFileSync(join(targetDir, "index.prose.md"), "---\nname: release-readiness\nkind: system\n---\n"); + writeFileSync( + join(targetDir, "release-readiness.test.prose.md"), + "---\nname: release-readiness-test\nkind: test\nsubject: release-readiness\n---\n", + ); + } + return 0; + }, + }), + }); + + expect(exitCode).toBe(0); + expect(seen).toEqual([ + "prose write output_mode: source-package-and-files apply: true target_path: src/release-readiness post_apply_action: host-will-run-root run_state: filesystem terminal_summary: required interactive: false request: 'draft release readiness'", + "prose run src/release-readiness/index.prose.md", + ]); + } finally { + rmSync(cwd, { recursive: true, force: true }); + } + }); + + it("does not run pre-existing unchanged tests in the write target", async () => { + const cwd = mkdtempSync(join(tmpdir(), "prose-write-existing-tests-")); + const io = memoryStreams(); + const seen: string[] = []; + const targetDir = join(cwd, "src", "release-readiness"); + + try { + mkdirSync(targetDir, { recursive: true }); + writeFileSync( + join(targetDir, "release-readiness.test.prose.md"), + "---\nname: release-readiness-test\nkind: test\nsubject: release-readiness\n---\n", + ); + + const exitCode = await runForwardedProseCommand({ + command: "write", + argv: ["--out", "src/release-readiness", "--apply", "draft release readiness", "--harness", "mock"], + cwd, + env: {}, + stdout: io.streams.stdout, + stderr: io.streams.stderr, + harnessFactory: () => ({ + name: "mock", + async run(prompt) { + seen.push(prompt); + if (prompt.startsWith("prose write ")) { + writeFileSync(join(targetDir, "index.prose.md"), "---\nname: release-readiness\nkind: system\n---\n"); + } + return 0; + }, + }), + }); + + expect(exitCode).toBe(0); + expect(seen).toEqual([ + "prose write output_mode: source-package-and-files apply: true target_path: src/release-readiness post_apply_action: none run_state: filesystem terminal_summary: required interactive: false request: 'draft release readiness'", + ]); + } finally { + rmSync(cwd, { recursive: true, force: true }); + } + }); + + it("fails when a repair pass deletes an initially generated failing test", async () => { + const cwd = mkdtempSync(join(tmpdir(), "prose-write-test-deleted-")); + const io = memoryStreams(); + const seen: string[] = []; + const targetDir = join(cwd, "src", "release-readiness"); + let writePasses = 0; + + try { + const exitCode = await runForwardedProseCommand({ + command: "write", + argv: [ + "--out", + "src/release-readiness", + "--apply", + "--test-iterations=2", + "draft release readiness", + "--harness", + "mock", + ], + cwd, + env: {}, + stdout: io.streams.stdout, + stderr: io.streams.stderr, + harnessFactory: () => ({ + name: "mock", + async run(prompt, runOptions) { + seen.push(prompt); + if (prompt.startsWith("prose write ")) { + writePasses += 1; + mkdirSync(targetDir, { recursive: true }); + writeFileSync(join(targetDir, "index.prose.md"), "---\nname: release-readiness\nkind: system\n---\n"); + const testPath = join(targetDir, "release-readiness.test.prose.md"); + if (writePasses === 1) { + writeFileSync(testPath, "---\nname: release-readiness-test\nkind: test\nsubject: release-readiness\n---\n"); + } else { + rmSync(testPath, { force: true }); + } + return 0; + } + if (prompt.startsWith("prose test ")) { + runOptions.stderr.write("expected release signal\n"); + return 7; + } + return 0; + }, + }), + }); + + expect(exitCode).toBe(1); + expect(seen.filter((prompt) => prompt.startsWith("prose test "))).toHaveLength(1); + expect(seen.filter((prompt) => prompt.startsWith("prose write "))).toHaveLength(2); + expect(io.stderr).toContain( + "Generated test target disappeared after repair: src/release-readiness/release-readiness.test.prose.md", + ); + } finally { + rmSync(cwd, { recursive: true, force: true }); + } + }); + + it("bounds generated test repair passes and keeps forwarded write side effects off", async () => { + const cwd = mkdtempSync(join(tmpdir(), "prose-write-test-repair-")); + const io = memoryStreams(); + const seen: string[] = []; + const targetDir = join(cwd, "src", "release-readiness"); + + try { + const exitCode = await runForwardedProseCommand({ + command: "write", + argv: [ + "--out", + "src/release-readiness", + "--apply", + "--test-iterations=3", + "draft release readiness", + "--harness", + "mock", + ], + cwd, + env: {}, + stdout: io.streams.stdout, + stderr: io.streams.stderr, + harnessFactory: () => ({ + name: "mock", + async run(prompt, runOptions) { + seen.push(prompt); + if (prompt.startsWith("prose write ")) { + mkdirSync(targetDir, { recursive: true }); + writeFileSync(join(targetDir, "index.prose.md"), "---\nname: release-readiness\nkind: system\n---\n"); + writeFileSync( + join(targetDir, "release-readiness.test.prose.md"), + "---\nname: release-readiness-test\nkind: test\nsubject: release-readiness\n---\n", + ); + return 0; + } + if (prompt.startsWith("prose test ")) { + runOptions.stderr.write("expected release signal\n"); + return 7; + } + return 0; + }, + }), + }); + + expect(exitCode).toBe(7); + expect(seen.filter((prompt) => prompt.startsWith("prose test "))).toHaveLength(3); + expect(seen.filter((prompt) => prompt.startsWith("prose write "))).toHaveLength(3); + expect(seen[1]).toBe("prose test src/release-readiness/release-readiness.test.prose.md"); + expect(seen[2]).toContain("Repair the generated OpenProse source under `src/release-readiness`"); + expect(seen[2]).toContain("Original request: draft release readiness"); + expect(seen[2]).toContain("Captured test output"); + expect(seen[2]).toContain("do not perform optional giving-back, memory, or mycelium note side effects"); + } finally { + rmSync(cwd, { recursive: true, force: true }); + } + }); + it("does not start the write follow-up run when aborted after apply", async () => { const io = memoryStreams(); const seen: string[] = []; diff --git a/tools/cli/tests/prose/command-model.test.ts b/tools/cli/tests/prose/command-model.test.ts index 4e4a684f..37e740ec 100644 --- a/tools/cli/tests/prose/command-model.test.ts +++ b/tools/cli/tests/prose/command-model.test.ts @@ -1,8 +1,10 @@ import { describe, expect, test } from "vitest"; -import { CommandModelError, canonicalPrompt } from "../../src/prose/index.js"; +import { CommandModelError, canonicalPrompt, parseWriteCommand } from "../../src/prose/index.js"; describe("command model", () => { + const writeUsage = "prose write [--out ] [--apply] [--run] [--test-iterations <0-3>] [request...]"; + const supportedCases: Array<[Parameters, string]> = [ [["compile", []], "prose compile"], [["compile", ["."]], "prose compile ."], @@ -18,6 +20,9 @@ describe("command model", () => { ], [["write", ["draft a release readiness responsibility"]], "prose write output_mode: source-package-only apply: false post_apply_action: none run_state: in-context terminal_summary: required interactive: false request: 'draft a release readiness responsibility'"], [["write", ["--out", "src/release-readiness", "draft a release readiness responsibility"]], "prose write output_mode: source-package-only apply: false target_path: src/release-readiness post_apply_action: none run_state: in-context terminal_summary: required interactive: false request: 'draft a release readiness responsibility'"], + [["write", ["--test-iterations=0", "draft a release readiness responsibility"]], "prose write output_mode: source-package-only apply: false post_apply_action: none run_state: in-context terminal_summary: required interactive: false request: 'draft a release readiness responsibility'"], + [["write", ["--test-iterations", "1", "draft a release readiness responsibility"]], "prose write output_mode: source-package-only apply: false post_apply_action: none run_state: in-context terminal_summary: required interactive: false request: 'draft a release readiness responsibility'"], + [["write", ["--out", "src/release-readiness", "--run", "--test-iterations=3", "draft a release readiness responsibility"]], "prose write output_mode: source-package-and-files apply: true target_path: src/release-readiness post_apply_action: host-will-run-root run_state: filesystem terminal_summary: required interactive: false request: 'draft a release readiness responsibility'"], [["write", ["--out", "src/release-readiness", "--apply", "draft a release readiness responsibility"]], "prose write output_mode: source-package-and-files apply: true target_path: src/release-readiness post_apply_action: none run_state: filesystem terminal_summary: required interactive: false request: 'draft a release readiness responsibility'"], [["write", ["--out=src/release-readiness", "--run", "draft a release readiness responsibility"]], "prose write output_mode: source-package-and-files apply: true target_path: src/release-readiness post_apply_action: host-will-run-root run_state: filesystem terminal_summary: required interactive: false request: 'draft a release readiness responsibility'"], [["write", ["--out", "src/vulnerability-detection", "--run", "a vulnerability detection system that uses lessons from https://blog.cloudflare.com/cyber-frontier-models/"]], "prose write output_mode: source-package-and-files apply: true target_path: src/vulnerability-detection post_apply_action: host-will-run-root run_state: filesystem terminal_summary: required interactive: false request: 'a vulnerability detection system that uses lessons from https://blog.cloudflare.com/cyber-frontier-models/'"], @@ -48,6 +53,13 @@ describe("command model", () => { ); }); + test("parses write test iteration bounds without passing them to prose-author", () => { + expect(parseWriteCommand(["draft"]).testIterations).toBe(1); + expect(parseWriteCommand(["--test-iterations=0", "draft"]).testIterations).toBe(0); + expect(parseWriteCommand(["--test-iterations", "1", "draft"]).testIterations).toBe(1); + expect(parseWriteCommand(["--test-iterations=3", "draft"]).testIterations).toBe(3); + }); + const validationCases: Array<[Parameters, string, string]> = [ [["compile", ["one", "two"]], "Unexpected argument 'two'", "prose compile [path] [--out ]"], [["compile", ["--out"]], "Missing value for --out", "prose compile [path] [--out ]"], @@ -57,19 +69,27 @@ describe("command model", () => { [["run", []], "Missing required argument ", "prose run [inputs...]"], [["run", ["system.md"]], "Expected ", "prose run [inputs...]"], [["run", ["script.prose"]], "Expected ", "prose run [inputs...]"], - [["write", []], "Pass text arguments or pipe stdin", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", [" "]], "Pass text arguments or pipe stdin", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--interactive", "draft"]], "does not support interactive flags", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--no-interactive", "draft"]], "does not support interactive flags", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--apply", "draft"]], "require --out ", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--run", "draft"]], "require --out ", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--out"]], "Missing value for --out", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--out=", "draft"]], "Missing value for --out", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--out", "/tmp/source", "draft"]], "root-relative path", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--out", "../source", "draft"]], "inside the OpenProse root", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--out", "source.md", "draft"]], "end in .prose.md", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--out", "src/source.txt", "draft"]], "end in .prose.md", "prose write [--out ] [--apply] [--run] [request...]"], - [["write", ["--out", "one", "--out", "two", "draft"]], "Duplicate option", "prose write [--out ] [--apply] [--run] [request...]"], + [["write", []], "Pass text arguments or pipe stdin", writeUsage], + [["write", [" "]], "Pass text arguments or pipe stdin", writeUsage], + [["write", ["--interactive", "draft"]], "does not support interactive flags", writeUsage], + [["write", ["--no-interactive", "draft"]], "does not support interactive flags", writeUsage], + [["write", ["--test-iterations"]], "Missing value for --test-iterations", writeUsage], + [["write", ["--test-iterations", "--apply", "draft"]], "Missing value for --test-iterations", writeUsage], + [["write", ["--test-iterations=", "draft"]], "Missing value for --test-iterations", writeUsage], + [["write", ["--test-iterations=-1", "draft"]], "between 0 and 3", writeUsage], + [["write", ["--test-iterations=4", "draft"]], "between 0 and 3", writeUsage], + [["write", ["--test-iterations=1.5", "draft"]], "between 0 and 3", writeUsage], + [["write", ["--test-iterations=lots", "draft"]], "between 0 and 3", writeUsage], + [["write", ["--test-iterations=1", "--test-iterations=0", "draft"]], "Duplicate option", writeUsage], + [["write", ["--apply", "draft"]], "require --out ", writeUsage], + [["write", ["--run", "draft"]], "require --out ", writeUsage], + [["write", ["--out"]], "Missing value for --out", writeUsage], + [["write", ["--out=", "draft"]], "Missing value for --out", writeUsage], + [["write", ["--out", "/tmp/source", "draft"]], "root-relative path", writeUsage], + [["write", ["--out", "../source", "draft"]], "inside the OpenProse root", writeUsage], + [["write", ["--out", "source.md", "draft"]], "end in .prose.md", writeUsage], + [["write", ["--out", "src/source.txt", "draft"]], "end in .prose.md", writeUsage], + [["write", ["--out", "one", "--out", "two", "draft"]], "Duplicate option", writeUsage], [["inspect", []], "Missing required argument ", "prose inspect "], [["lint", ["system.md"]], "Expected ", "prose lint "], [["preflight", ["system.md"]], "Expected ", "prose preflight "], diff --git a/tools/cli/tests/skills/open-prose.test.ts b/tools/cli/tests/skills/open-prose.test.ts index 7bf8d66a..0149fc73 100644 --- a/tools/cli/tests/skills/open-prose.test.ts +++ b/tools/cli/tests/skills/open-prose.test.ts @@ -81,6 +81,17 @@ describe("OpenProse skill checks", () => { expect(normalized).toContain("Authoring success must not depend on optional memory or note writes"); }); + it("documents write test iterations as a host-adapter macro that unsupported session routers must reject", () => { + const source = readFileSync(join(repoRoot, "skills/open-prose/SKILL.md"), "utf8"); + const normalized = source.replace(/\s+/g, " "); + + expect(source).toContain("--test-iterations"); + expect(normalized).toContain("runs ordinary `prose test` for generated `kind: test` files"); + expect(normalized).toContain("reject an explicit `--test-iterations` option before authoring"); + expect(normalized).toContain("must not pass the test-iteration macro into `prose-author`"); + expect(normalized).toContain("must not perform optional giving-back, memory, or mycelium note side effects"); + }); + it("builds a bootstrap prompt with the full skill text and skill root instructions", () => { const bootstrap = buildOpenProseSkillBootstrapPrompt({ skillPath: "/home/prose/.agents/skills/open-prose/SKILL.md",