Grading CLI observability, caching & emit-skill (Memo 110) - #99
Merged
Conversation
Phase 2 (Memo 110): cache control on the CLI. - thread --force through gradingDeterministic -> DataPretest.run - surface fromCache + dataAt in the deterministic result - new `grading reload <ns|ns/schema>`: re-fetch + rewrite test-N.json only (force), decoupled from grading (no _gradings/grade.json writes) - tests for force-threading, data-stamp surfacing, reload routing Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 (Memo 110): the emit returns a single self-contained Emit-Skill. - #buildEmitSubstitutions feeds real repo-relative schema paths + tool/ namespace names into the composed area prompts (no torso) - #buildEmitSkill assembles ONE skill text: self-describing header, the bundled READY-stage areas, and the Task-ID + exact --consume-scores return command IN THE TEXT (Kap 10.1); gated stage-2 areas named for the follow-up emit - maxTurns configurable (--max-turns, default 25; was hardcoded) - tests for emit-skill text, maxTurns flow + malformed-value guard Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4 (Memo 110): observability (Befund B). - #emitProgress ticks per-schema/per-tool to STDERR during the slow deterministic/emit/reload runs; --quiet silences it - stdout machine JSON is untouched (piped `... | jq` stays pure) - printDeterministicSummary: concise human summary to stderr, gated by !quiet && !json; no machine key dropped (rollup/proof consumers depend on them) — summary is additive - tests: stderr ticks vs --quiet, summary gating Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 5 (Memo 110, I-4): a `both`-classified area carries a free deterministic gate AND an LLM round, so nextAction lists it in deterministicNow AND the non-det area-set. Doctor test reflects the corrected below-deterministic-green behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #98. Pins flowmcp-grading v2.4.0 (FlowMCP/flowmcp-grading#60).
Phases (cli side)
--forcethreads the cache bypass intoDataPretest.run;fromCache+dataAtsurfaced in the deterministic result; newgrading reload <ns|ns/schema>(re-fetch + rewrite test-N.json only, no_gradings/grade writes).#buildEmitSubstitutions(repo-relative paths) +#buildEmitSkillassemble ONE self-contained skill text (ready-stage areas, real paths, inline schema, Task-ID +--consume-scorescommand IN the text); gatednamespace-*named for the follow-up emit;--max-turnsconfigurable.#emitProgress→ stderr per-schema/tool ticks;--quiet;printDeterministicSummary(stderr, additive, no machine key dropped; stdout JSON pure).both-classified areas route to deterministicNow AND the non-det area-set.Tests: 134/134 grading-suite pass against the published grading v2.4.0; full suite parity with main (pre-existing non-grading suites unaffected).
🤖 Generated with Claude Code