PromptDiff

PromptDiff treats prompts like code: compare revisions, call out risky behavior changes, and leave a tidy local report that reviewers can trust.

It is a small TypeScript CLI for developers and agent operators who want deterministic prompt review evidence without SaaS dashboards, telemetry, or hidden network calls.

Why it exists

Prompt edits can quietly change instruction hierarchy, tool access, and output contracts. Those changes are easy to miss in a normal text diff but can break agents, parsers, and safety assumptions. PromptDiff gives those changes names.

Install

npm install
npm run build

For local development, run the built CLI directly:

node dist/cli.js --help

After publishing/installing globally, use:

promptdiff --help

Quick start

npm run build
node dist/cli.js compare examples/prompts/v1.md examples/prompts/v2.md --out prompt-risk.md
node dist/cli.js compare examples/prompts/v1.md examples/prompts/v2.md --format json
node dist/cli.js check examples/prompts/*.md --rules examples/rules.json --fail-on high

Commands

compare

Compares two prompt/template revisions and produces stable Markdown or JSON.

promptdiff compare old.md new.md --format markdown --out report.md --fail-on high
promptdiff compare old.json new.json --format json

It classifies:

• instruction changes, including dangerous “ignore/bypass/override” language
• removed safety/refusal/secret-handling guardrails
• tool surface changes such as shell, browser, network, delete, database, or email access
• output-contract changes such as JSON/schema/format shifts
• secret-like values, redacted before output by default

check

Checks one or more prompt files against a simple JSON rules file.

{
  "requiredPhrases": ["protect customer secrets"],
  "forbiddenPhrases": ["ignore previous instructions"],
  "requireSections": ["Role", "Instructions", "Output Contract"],
  "maxSeverity": "high"
}

promptdiff check prompts/*.md --rules promptdiff.rules.json --fail-on high

Exit code 2 means the configured quality gate failed. Exit code 1 means a command/runtime error.

Supported inputs

• Markdown
• plain text
• JSON with stable key ordering
• JSONL / NDJSON with stable key ordering per line

Safety model

• Local-first by design.
• No telemetry.
• No external network calls.
• No hidden file writes; files are written only when --out is provided.
• Redaction is enabled by default for common API key, token, password, and credential patterns.
• Reports use deterministic timestamps (deterministic-local) to avoid noisy snapshots.

Limitations

PromptDiff is a deterministic heuristic tool, not an LLM judge. It is deliberately conservative and review-friendly. It will miss subtle semantic changes, and it may flag harmless text that looks like risky instruction/tool/output-contract language.

Examples

Fixtures live in examples/:

npm run build
node dist/cli.js compare examples/prompts/v1.md examples/prompts/v2.md
node dist/cli.js check examples/prompts/safe.md --rules examples/rules.json
npm run smoke

For a reviewer-facing walkthrough, see docs/tutorials/review-agent-tool-expansion.md. It demonstrates a prompt revision that expands browser and shell tool language, removes an explicit secret-handling guardrail, and changes the output contract.

For a release-gate style demo that writes review artifacts to a temporary directory, run:

npm run build
bash examples/release-gate-demo.sh

The demo intentionally expects the compare command to exit 2 because --fail-on high catches the risky prompt expansion. See docs/tutorials/prompt-release-gate.md for the walkthrough and docs/promo/release-gate-social-hooks.md for grounded launch copy.

To generate the walkthrough evidence in one command:

npm run build
bash demo/review-tool-expansion.sh

The script writes Markdown and JSON reports under demo/output/. A promotion-ready launch note draft lives in docs/promo/launch-note.md.

You can also run the same scenario as a fixture-backed demo script:

bash demo/tool-expansion-review.sh

The script writes Markdown, JSON, and rules-check outputs under /tmp/promptdiff-demo or $TMPDIR/promptdiff-demo.

For a compact demo that writes Markdown and JSON review reports from the checked-in v1 and v2 fixtures:

bash demo/review-output-contract-change.sh

See docs/tutorials/review-output-contract-change.md for the full walkthrough.

To capture a rules-gate pass and an expected high-risk failure:

bash demo/rules-gate-smoke.sh

See docs/tutorials/rules-gate-smoke.md for the generated reports and review angle.

To generate the tool-expansion review artifacts in one command:

bash demo/run-tool-expansion-review.sh

Development

npm install
npm run check
npm test
npm run build
npm run smoke
bash scripts/validate.sh

Contributing

Issues and small PRs are welcome. Please keep changes deterministic, local-first, and covered by fixtures/tests. If you add a new risk classifier, add one example prompt pair and one test that proves the report stays stable.

License

MIT

Release Readiness

Use the checked-in scripts before opening or publishing a release:

npm run check
npm test
npm run build
npm run smoke
npm run package:smoke
npm run release:check

The package smoke uses npm pack --dry-run and fails if the CLI, library entrypoint, license, security policy, changelog, or contribution guide would be missing from the published tarball.