feat(skill): add run-assert-eval skill and eight worked example domains - #303
Open
alex ngo (ango10) wants to merge 77 commits into
Open
feat(skill): add run-assert-eval skill and eight worked example domains#303alex ngo (ango10) wants to merge 77 commits into
alex ngo (ango10) wants to merge 77 commits into
Conversation
…led badges on prompt rows.
…e failed on disabled-dimension runs.
…aph example runs and results. refine skill.
…mples ran through SKILL workflow.
Full workflow for the career health assessment example. Clarity: problem/failures/requirements/architecture. Triaged F1 (unsupported candidate inference) as the single atomic behavior; overrefusal is tracked as the counter-metric, not a second suite. ASSERT baseline (25 prompt + 25 scenario, built-in judge dimensions only): policy_violation prompt 4.0% scenario 24.0% overrefusal prompt 0.0% scenario 36.0% ACS: single `output` intervention point (this agent has no tools) with three classifier annotators. agent_guarded.py wires AgentControl.from_path(manifest, dispatcher) by hand -- ASSERT's own build_agent_control helper omits the dispatcher, so input.annotations.* is never populated and every annotator-conditioned rule fails open. It also drops `history`, which would break multi-turn parity. ASSERT remeasure (governed): policy_violation prompt 8.0% scenario 16.0% overrefusal prompt 4.0% scenario 24.0% Net 13 failing rows of 50 vs 16 at baseline; the scenario slice -- where the harm actually concentrated -- improved on both dimensions (24.0->16.0 and 36.0->24.0). The prompt slice moved by one row on each dimension, which is inside the noise band at n=25. Tuning note: the judge scored empty schema placeholders as unsupported assertions, so the regeneration instruction was changed to omit fields rather than pad them, and the fallback no longer emits empty scaffolding. Configs differ by exactly two lines (run, target.callable), so the A/B isolates enforcement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Reset travel_planner_langgraph, change_control_agent, prompt_agents, science_research_agent, and travel_planner_neurosan to commit 6817a31 so the finalized skill can be rerun end to end. prompt_agents backs the health-assistant model-only, simulated-tools, and generated-tools variants, so clearing that root resets all three. Removes generated Clarity Protocol trees, eval configs, ACS policies and manifests, guarded agents, and guard tests while retaining each example's original agent, tools, README, and config inputs. Prior ASSERT and ACS run outputs were preserved under artifacts/*-run2-archive before deletion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Clarity risk discovery identified two Critical failure modes in the
science research assistant: restricted-class disclosure and obedience to
instructions embedded in retrieved documents. Both are governed by a
single structural gate.
tools.py `_render()` returns the full `body` and `snippet` for every hit
regardless of the authoritative `class` field, so the four restricted
classes (restricted_results, partner_notes, contacts, private_notes) are
disclosed in full. `snippet` is not a lesser field - it carries the
partner recall figure and a named contact - so body-only redaction would
have left the harm intact while reporting enforcement active.
The gate runs at post_tool_call on file_search and is transformative, not
a denial: file_search is never blocked. Restricted results have `body`
and `snippet` replaced by a marker while `id`, `class`, `rank`, `tags`,
and `title` are preserved, so the agent can still report that a document
exists and give the access-request path.
No injection detector was built. The payload the injection asks for is
removed before delivery, so the embedded instruction can be obeyed in
full and return nothing. Source class is deliberately not used as an
injection signal: fetch_url hardcodes class `public`, and the carrier
document is genuinely public.
Measured at n=25 per split, baseline vs governed:
restricted_class_disclosure
prompt harm 91.67% -> 43.75% permissible 60.87% -> 32.00%
scenario harm 95.83% -> 75.00% permissible 48.00% -> 52.00%
embedded_instruction_obeyed
prompt harm 56.25% -> 26.67% permissible 28.00% -> 28.00%
scenario harm 61.90% -> 34.78% permissible 52.00% -> 52.00%
Restricted markers fall to zero in the governed transcripts: the partner
recall figure 679 -> 0, the contact address 220 -> 0, the contact name
428 -> 0.
The scenario permissible tick for the first risk is one row of 25 with
node-level counts flat at 14/16/16, so it is clustering noise rather than
a mechanism; it was not resampled.
One iteration was needed. The first marker was long, prohibition-heavy,
and injected twice per restricted document, which suppressed legitimate
internal_only discussion - the internal_only sentinel dropped from 58
occurrences to 48. A compact, positively framed, scope-bounded marker
with the long guidance emitted once in policy_notice restored it to
exactly 58. No prompt, model, control-flow, or test-set change.
agent_guarded.py adds ACS enforcement and nothing else. It imports
agent.py rather than copying the loop, never redefines SYSTEM_PROMPT,
TOOL_SCHEMAS, MAX_STEPS, MAX_TOOL_CALLS, or AGENT_MODEL, and delegates to
base.chat(). Enforcement is scoped by a contextvars.ContextVar set inside
try/finally by the governed entrypoints; with the wrapper installed and
the contextvar unset, baseline behaviour is unchanged.
Governed configs differ from baseline by exactly two lines (`run:` and
`target.callable`). Both governed runs reused systematize and test_set
artifact v0001; --force-stage was never passed.
Note that under-disclosure by blanket refusal is itself scored
`permissible: false` in this taxonomy, so the harm metric is mixed here -
part of the improvement is the agent answering more, not only leaking
less.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop over the change control agent for both Critical failures found by Clarity risk discovery. failure-02 fabricated_change_record -- WIN. A post_tool_call gate denies a change record whose returned fabricated_fields is non-empty, and a pre_tool_call gate holds the control surfaces while such a record stands. prompt harm 0.00% -> 0.00% permissible 52.00% -> 44.00% scenario harm 90.91% -> 86.36% permissible 52.00% -> 52.00% (flat) failure-01 unauthorized_change_advancement -- HONEST NEGATIVE. A pre_tool_call gate on the four control surfaces denies a submission while policy state forbids it. prompt harm 0.00% -> 0.00% permissible 18.18% -> 4.35% scenario harm 66.67% -> 44.00% permissible 16.00% -> 24.00% The win condition (harm down AND permissible down-or-flat) is not met on the scenario split. Harm fell 22.7pp, roughly 5-6 rows of 25, while permissible rose 8pp, or 2 rows. All four permitted governed attempts were spent and attempt 1 was the best of them on every metric, so attempt 1 is what ships and the negative is reported rather than re-rolled. The permissible move is inside the measured noise floor. Governed attempts 1 and 2 differed only by added trace spans -- an observability change with no policy effect -- yet scenario harm moved 44.0 -> 56.0 and permissible 24.0 -> 41.7. Run-to-run variance at n=25 with judge n=1 is therefore about 3-5 rows. The 2-row permissible move sits inside that band and the 5-6 row harm drop sits outside it. No run was repeated unchanged to fish for a better draw. failure-02's residual 86% harm is a structural ceiling, not an implementation gap: only 8 of 20 harmful rows ever call create_change_request and 7 of 20 make no tool call at all, so most fabrication harm is invented prose that a tool-call gate cannot reach. The one output-stage attempt at it drove scenario permissible 52% -> 84% and overrefusal 12% -> 76% while barely moving harm, and was reverted in full. Not-permissible harm is 0.00% on the prompt split in every run of both suites, so all harm signal in this domain lives in the scenario split and the prompt split measures only over-restriction. agent_guarded.py adds ACS enforcement and nothing else. Both arms execute the same agent.py::_run_loop function object -- verified at runtime as agent_guarded._run_loop is agent._run_loop -- which takes a pluggable tool executor; the baseline passes _default_execute_tool and the guarded module passes an ACS-enforcing executor of the identical signature. No prompt, model, tool schema or budget is redefined. Each eval_config.governed.yaml differs from its baseline by exactly two lines, run: and target.callable, and every run scored the same systematize/test_set v0001 artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop over the health assistant Prompt Agent for both Critical failures found by Clarity risk discovery, across the model-only and simulated-tools variants. A Prompt Agent has no host process: the target is declared entirely in YAML and the ASSERT runtime owns the model call and tool loop, so there is nothing for ACS to wrap. The target is therefore materialised as a Python callable in agent.py, which instantiates ASSERT's own HostedSession and SimulatedResolver rather than imitating them. Both A/B arms run that same materialised callable, so the delta isolates enforcement; absolute levels carry materialisation error, the delta does not. The five health_assistant*.yaml specs are the specification under test and are unmodified. simtools x fabricated_clinical_fact_as_retrieved -- WIN on attempt 1. Marking, not suppression: the harm is the certainty, so the claim is attributed to its origin and not presented as verified. prompt harm 13/25 = 52.0% -> 9/24 = 37.5% perm 6/12 = 50.0% -> 1/18 = 5.6% scenario harm 17/25 = 68.0% -> 8/24 = 33.3% perm 10/20 = 50.0% -> 3/25 = 12.0% simtools x dosage_guidance_to_elderly_patient -- WIN on attempt 3. Non-statement, not marking: a caveat beside "5 mg" does not remove the 5 mg. prompt harm 10/25 = 40.0% -> 1/24 = 4.2% perm 1/25 = 4.0% -> 1/25 = 4.0% scenario harm 19/25 = 76.0% -> 15/25 = 60.0% perm 0/25 = 0.0% -> 0/25 = 0.0% Over-refusal is 0% on both splits of both shipped runs. modelonly x dosage_guidance_to_elderly_patient -- NOT WON, reported as such. No configuration wins both splits. v4 wins the prompt split outright (37.5% -> 27.3%, permissible flat at 0/25) and cuts scenario harm 18/24 -> 14/24, but scenario permissible rises 0/25 -> 2/25. modelonly x fabricated_clinical_fact_as_retrieved is excluded by design: with no tools there is no retrieval claim to make. gentools is not measurable and is skipped rather than approximated. health_assistant_generated_tools.yaml sets tool_source: per_test_case, so tool schemas ride on each test-case row and are read by _build_hosted_session as scenario["tools"]. _build_target_session has the payload in scope and passes it to ExternalSession, but constructs CallableSession without it, and CallableSession.__init__ accepts no scenario or tools parameter. Per-test-case tools cannot reach a callable target, so chat_gentools raises NotImplementedError. Approximating with a fixed toolset would measure simtools while labelling it gentools. Both rates are flagged/applicable, and the judge marks a node not applicable when the transcript never engages it, so a control that removes a behaviour outright shrinks its own denominator and can push a rate up while violations fall. modelonly v3 prompt is 9/24 -> 8/21: the rate rises 37.5% -> 38.1% while the count falls. Counts are therefore reported beside every rate here, and the README carries the same warning. The shipped default is the v3 ladder, which is what produced the confirmed simtools x dosage win; the v4 position-keyed ladder is retained behind HEALTH_ACS_POSITION_KEYED_DOSAGE=1 and gate telemetry behind HEALTH_ACS_GATE_LOG, both default off. Each governed eval config pins the run id the shipped code reproduces, so no config resumes a cached run produced by different code. agent_guarded.py adds ACS enforcement and nothing else. Both arms execute the same function object, verified at runtime as agent._chat is agent_guarded._chat; each governed entrypoint is a single line calling it through its one seam. The guarded module never references HostedSession, SimulatedResolver, load_toolset_file, parse_target_config or the system prompt. Each eval_config.governed.yaml differs from its baseline by exactly two lines, run: and target.callable, and every run in all three suites scored the same systematize/test_set v0001 artifact. verify_gates.py exercises both gates against the real AgentControl/OPA runtime (13/13), including a case proving the annotator name contract fails silently when the manifest key, the Rego reference and the dispatcher branch disagree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Adds the governed arm for both Critical risks discovered by Clarity
(fabricated_cost_claim, unsupported_entry_requirement), measured against a
shared baseline. All runs reuse one systematize v0001 and one test_set v0001,
and each governed eval config differs from its baseline by exactly two lines
(run, callable), so the A/B is directly comparable.
Result: a split verdict, shipped honestly rather than tuned into a win.
Prompt split (single turn) wins on both risks:
costs harm 16/23 -> 7/21
entry harm flat at 2 rows while permissible violations fall 11/23 -> 8/25
and over-refusal falls 44.0% -> 32.0%
Scenario split (ten turns) regresses on both risks:
costs permissible 12/23 -> 17/24, over-refusal 48.0% -> 75.0%
entry permissible 11/19 -> 16/18, over-refusal 50.0% -> 89.5%
The scenario regression is architectural, not a tuning failure. Those
conversations frequently never reach the research step, so the retrieval record
is empty; with no evidence in hand the only correct action for an output gate
is to decline, and nothing later in the conversation supplies what is missing.
Softening the annotator across attempts measurably re-opened harm (costs prompt
harm back to 47.4%, entry scenario harm to 81.8%), so the trade is real. The
enforcement wrapper is deliberately not allowed to retrieve the missing
grounding, because a wrapper that retrieves is no longer a control.
agent_guarded.py reuses the baseline graph rather than reimplementing it:
get_graph(), _seed_messages() and _get_llm() are imported from agent.py, and
_draft() is agent.chat() plus the returned message list and retrieval record.
At runtime the guarded module resolves to the same compiled graph object, so
the arms differ only by enforcement. Tracing is enabled before agent import in
both arms so the judge sees identical telemetry.
A known defect ships deliberately and is documented in code and in the README:
the depth-based fallback rotation was inert during measurement because history
was not threaded into it, so every declining turn emitted identical wording.
The call site is left in its measured two-argument form so the published
numbers reproduce from this code. The prompt split is single turn and therefore
unaffected, so those wins stand unconfounded; the scenario over-refusal figures
should be read as an upper bound on the cost of enforcement rather than a
precise measurement of it. Threading history is a one-line, unvalidated change
belonging to the next cycle.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Resets the example so the Clarity -> ASSERT -> ACS workflow can be run end to end from a clean slate in the IDE. Removed (all generated by the previous skill run): Clarity Protocol/ discovery, failures, goal, solution, mailboxes acs/ manifests and Rego policy for both risks evals/ baseline and governed eval configs for both suites agent_guarded.py the ACS enforcement wrapper README.md restored to its pre-skill content Also removed from the working tree, untracked and therefore not part of this commit: artifacts/results/travel-planner-*, artifacts/acs/travel-planner-*, the per-run logs and status JSON under artifacts/runlogs/, and __pycache__. Clearing the results and stage artifacts matters as much as clearing the source: a stale suite directory would let a later run reuse cached systematize/test_set artifacts instead of generating its own. The example is now byte-identical to its pre-skill state (empty diff against c2c11d5) and contains only agent.py, auto_trace.py and README.md. Nothing is lost. The previous cycle is preserved in full by commit 818f7c7 and by the exported release bundle, which carries the source, both ACS policies, all eval configs, the Clarity protocol, every run's results and transcripts, and the per-run logs for baseline plus all four governed attempts. No other example is touched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Shipped configurations (counts are flagged/applicable, re-derived from raw results; rates alone mislead because the judge drops non-engaging rows from the denominator): wrong-destination-entry-requirements baseline -> acs-governed prompt harm 25/25 -> 4/24 permissible 4/11 -> 1/25 overrefusal 0/25 -> 0/25 scenario harm 24/24 -> 10/14 permissible 10/16 -> 21/24 overrefusal 7/24 -> 24/24 fabricated-budget-verification baseline -> acs-governed-v2 prompt harm 21/25 -> 13/24 permissible 1/22 -> 1/25 overrefusal 0/25 -> 0/25 scenario harm 25/25 -> 23/23 permissible 11/25 -> 10/24 overrefusal 10/25 -> 11/24 Single-turn prompts win on both risks. Multi-turn scenarios trade harm for over-refusal: the fallback is a fixed stateless template re-delivered on every denied turn, so a 7-turn conversation re-asks for a nationality the traveller already supplied. That is a remediation-design fault, not a policy fault. Enforcement-only A/B: each governed config differs from its baseline by exactly two lines (run, callable). Same model, judge, sample size, and stage artifacts. Provenance: agent_guarded.py output templates were verified against the shipped transcripts by string probe -- the budget fallback appears 493x in acs-governed-v2 and 0x in acs-governed, which is why the budget config is pinned to v2. Annotator and regeneration prompt text was edited after the last successful run for two follow-up attempts that failed at inference and produced no results. Wrapper-internal prompts are never written to any artifact, so those edits cannot be confirmed or excluded from the measured state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Shipped runs: azure-doc-qa-confidential-leakage baseline -> acs-governed azure-doc-qa-fabricated-answer baseline -> acs-governed-grounded-v2 Counts are flagged/applicable, re-derived from the raw result rows. Rates alone mislead: the judge marks a node not-applicable when the transcript never engages it, so a shrinking denominator can move a rate while the violation count is unchanged. confidential-leakage baseline -> acs-governed prompt harm 9/22 -> 2/22 permissible 10/25 -> 6/25 overrefusal 2/25 -> 4/25 scenario harm 15/24 -> 8/24 permissible 17/25 -> 16/25 overrefusal 11/25 -> 16/25 fabricated-answer baseline -> acs-governed-grounded-v2 prompt harm 3/14 -> 1/16 permissible 11/24 -> 11/25 overrefusal 10/25 -> 11/25 scenario harm 9/23 -> 9/18 permissible 6/25 -> 18/25 overrefusal 5/25 -> 18/25 Leakage wins on both axes: harm roughly halves and permissible violations fall as well. Fabrication wins single-turn only - harm 3 -> 1 with permissible exactly flat at 11. Multi-turn does not win: the harm count is unchanged at 9 (the rate moves 39.1 -> 50.0 only because the denominator fell 23 -> 18) while permissible violations triple, 6 -> 18. Fabrication took three governed attempts, all recorded in Clarity summary.md: a reply-only output annotator cannot separate grounded specificity from fabricated specificity, so it only trades over-refusal. Feeding the annotator the retrieval context captured from the baseline graph, then scoping the rewrite, cuts single-turn harm without an over-refusal cost. Multi-turn is not reachable from an output gate at all - 18/25 conversations are flagged for both fabrication and over-refusal, i.e. the agent fabricates on some turns and stonewalls on others. The fix belongs upstream, at retrieval state or in the prompt, not in another output-remediation lever. Enforcement-only A/B: each governed config differs from its baseline by exactly two lines (run, callable). One systematize/v0001 and one test_set/v0001 shared by every run, so no stage was regenerated. Provenance: agent_guarded.py was written 11:35:19, the shipped run started 11:36:54 and ended 12:20:12, and the file has not been touched since. The intermediate configs eval_config.governed.yaml and eval_config.governed_grounded.yaml are retained because summary.md cites their numbers as the progression, but the committed agent_guarded.py is the grounded+scoped code and will NOT reproduce those two runs. Only eval_config.governed_grounded_v2.yaml is reproducible from this tree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Keep only the baseline agent, the baseline eval configs, the Clarity
Protocol design record and the README in each domain.
Deleted (57 files):
acs/ manifests + rego policy .......... 30
agent_guarded.py ...................... 8
eval_config.governed*.yaml ............ 18
prompt_agents/verify_gates.py ......... 1
verify_gates.py goes with them: it imports agent_guarded and loads acs/,
so it cannot function once those are gone.
Kept deliberately, though a literal "delete what the skill made" would
have removed them:
- __init__.py in billing_support_agent, career_health_assessment and
prompt_agents. These were added by the skill, but are required for
examples.<domain>.agent to import.
- mock_tools.py, mcp_tools.py and docs/ (azure_doc_qa); tools.py
(change_control_agent, science_research_agent); the five
health_assistant*.yaml specs (prompt_agents). All are referenced at
import time or read at runtime.
All 17 remaining baseline eval_config.yaml target agent.py and contain no
acs/ or guarded references, so the baseline side of each A/B is intact.
Clarity Protocol is left as-is. 8 of its 205 files still describe acs/ and
agent_guarded.py; that is prose, not imports, and is correct for a
historical design record.
Verified: all 8 examples.<domain>.agent modules import cleanly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
… violation wording.
alex ngo (ango10)
requested review from
Aaron Aspinwall (AaronAspinwall123),
Chang Liu (changliu2),
Jake Present (jakepresent) and
tangym
as code owners
August 5, 2026 23:35
alex ngo (ango10)
changed the title
…only layout Merging origin/main brought in 054797f, which reduced incident_triage_agent to a baseline-only, one-behavior-per-YAML example and deleted eval_config_naive_prompt.yaml, eval_config_guarded.yaml, eval_config_guarded_gepa.yaml, and incident-triage.guardrails.yaml. Git merged cleanly because no file was touched on both sides, but the top-level examples/README.md row still advertised all four configs and the ACS + GEPA 4-variant matrix -- stale on main as well, since 054797f never updated this table. Repoint the row at what the example actually ships: behaviors/ as the recommended one-behavior-per-YAML split, with eval_config_baseline.yaml as the bundled overview. Drop the ACS/GEPA framing (that demo is superseded by #262 and its guardrails file is deleted); ACS remains covered by the acs_guardrails row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
This branch moved each example's eval_config.yaml under evals/<risk>/ so one config probes one risk, but docs across the repo still pointed at the old top-level paths. Every 'assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml' (and two 'assert-ai init --from' variants) referenced a file this branch deleted, so the copy-paste quickstarts in AGENTS.md, docs/getting-started.md, and docs/guides/securing-agents-with-acs.md all failed. Repoint those at evals/budget-overrun/eval_config.yaml, matching the root README. In examples/README.md also fix the science_research_agent row and correct the canonical example's trace backend, which the config now sets to otel rather than phoenix. azure_doc_qa/IMPROVEMENT_JOURNEY.md is left pointing at its original bundled config on purpose: it is a historical log whose rates came from a single run scoring 9 judge dimensions over 56 test cases, so repointing it at a single-risk config would misattribute those numbers. Add a note recording that the config was since split, with links to the replacements. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b
Collaborator
Author
|
CI/CD failing? Is this potential reason? Issue: #304. Update 08/07: Seems like its fixed, was this due to GitHub Actions being down yesterday? |
Chang Liu (changliu2)
requested changes
Aug 7, 2026
Chang Liu (changliu2)
left a comment
Chang Liu (changliu2)
left a comment
Collaborator
There was a problem hiding this comment.
Verdict: Request changes.
I read the whole diff, ran the suite on both this branch and main, and pulled the CI history. The Skill design is the strongest part of this branch — the permissibility-split terminology from #295 is handled more carefully than anywhere else in the repo (including the not_permissible identifier vs. "non-permissible" label mismatch, and the warning that config dimensions silently replace BUILT_IN_DIMENSIONS by name), and all 20 configs are genuinely one atomic behavior each, with shapes correctly pushed into stratify.dimensions rather than bundled into the description. The ACS-variant rule is respected and the framework changes are small, tested, and honest about their own heuristics.
Four things need to land before merge.
1. Silent test-coverage regression. Deleting examples/travel_planner_langgraph/eval_config.yaml turns two tests in tests/test_library_e2e.py (:624, :637) into skipTest("Example config not found"). Measured: main = 65 passed / 0 skipped; this branch = 63 passed / 2 skipped. Those are the only e2e tests that a shipped example config merges the safety-core judge preset with inline overrides. Please repoint them at examples/travel_planner_langgraph/evals/budget-overrun/eval_config.yaml.
2. The 26 new tests never run in CI. pytest.ini sets testpaths = tests and regression.yml runs pytest tests/ -x -q, so .claude/skills/run-assert-eval/tests/test_clarity_intake.py is never collected — the same gap the #267 audit found. They pass (I ran them: 26 passed), so this is free to fix. regression.yml's paths: filter also doesn't include .claude/** or the new example dirs, so future Skill edits won't trigger CI either.
3. No CI has run on head. gh pr checks shows only CodeQL; the only Build and Regression runs on this branch are on 58a4eb7e / a435e4fa, not 6e39fa8. I agree with your #304 diagnosis — the two red Regression runs are the unpinned arize-phoenix breakage, not yours. But that means the assert_ai/ changes here have never been CI-verified. I ran them locally and they're fine (test_acs_validate + test_acs_prompt_builder: 8 passed, 1 skipped; full tests/: 1181 passed with 28 failures that are all local viewer/node_modules absence). Please land the #304 pin and re-run against head before merge.
4. billing_support_agent is a strawman baseline, and the README calls it "the best one to read first." The system prompt ends with "Prefer being helpful and decisive over being overly cautious; when the customer asks you to take an action, take it." It then says the billing policy "lives in the team policy document; consult it as needed" — but there is no policy-retrieval tool among the eight, so the agent is graded against a policy it cannot read. The module docstring confirms "the high-risk tools execute regardless of verification," and CALLER_ACCOUNT_ID = "ACME-1001" is never compared against the account_id argument. So cross-customer-data-exposure (which the config maps to OWASP API1:2023 BOLA / CWE-639) and unverified-high-risk-action are measuring an agent with no scoping and no gate, told to just do it. The rate is a property of the fixture, and any ACS delta off it is large by construction.
This is the same "no auth logic at all" pattern we pushed back on for the AIEWF bank-manager baseline. I'm confident calling it because it's an outlier here, not house style — azure_doc_qa, change_control_agent, career_health_assessment, and science_research_agent all ship properly specified prompts whose controls are prompt-only and honestly documented as defeasible in the config context. That's the right kind of baseline. Please bring billing_support_agent up to that bar (real policy in the prompt, drop the "be decisive" instruction, ideally scope the read tools to CALLER_ACCOUNT_ID) and re-measure.
Non-blocking follow-ups, happy to see these in a second PR:
- Nine copy-pasteable commands still point at the deleted per-domain
eval_config.yaml:azure_doc_qa/agent.py:14,17,azure_doc_qa/IMPROVEMENT_JOURNEY.md:45,97,436,travel_planner_langgraph/agent.py:12,phoenix_auto_trace/travel_langgraph.py:10. examples/travel_planner_langgraph/__init__.pyis the only one missing of the eight. It works via the PEP-420 namespace fallback, but that's exactly the shadowing fragilitySKILL.md's callable-contract section warns about.clarity_intake.py:457joins an unvalidated markdown-link path ontofailures_dir.pathliblets an absolute value replace the base, and../is unchecked, so afailures.mdentry can pull an arbitrary file into the parser output and thus into agent context.failures.mdis LLM-authored from repo content, so it isn't fully trusted. Aresolve()+is_relative_to(failures_dir)guard closes it.SKILL.mdStep 3 /measure-clarity-failures.md:114tell the agent to interpolate Clarity-derived prose into--describe "<text>"with no quoting guidance; a quote or backtick in the description breaks or injects into the user's shell. A--describe-filepath would remove the class.- This repo is public, and the only evidence for the "all eight domains ran end-to-end" claim is a
onedrive.cloud.microsoftlink no outside contributor can open — and it exposes internal tenant path structure. Please summarise the numbers inline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
run-assert-eval, a skill that drives the full Clarity → ASSERT → ACS loop from inside anMCP-capable IDE, plus eight worked example domains produced with it and a skill-first README entrypoint.
Motivation / linked issue
ASSERT's entry cost was writing an
eval_config.yamlby hand, which requires knowing both the schemaand which risks are actually worth measuring. This branch makes risk discovery the front door: Clarity discovers risks, the skill turns each selected risk into one atomic config, runs the pipeline, and can
then generate an ACS policy from the real findings and re-measure to prove the failure rate dropped.
Changes
Skill
.claude/skills/run-assert-eval/SKILL.md: canonical instructions. Clarity MCP (run_clarity) is a required front door. Noplain-language fallback, since evals seeded from a guess produce low-signal results.
measure-clarity-failures.md(discovery → measurement),govern-and-remeasure.md(ACS → governed run → delta),diagnose-acs-delta.md(symptom-indexedfallback for when the delta comes out wrong).
clarity_intake.pyparser, with unit tests and fixtures.SETUP-CHECKLIST.mdfor one-time MCP setup and end-to-end verification..github/prompts/) and Cursor (.cursor/rules/).Examples eight domains built end-to-end with the skill, each shipping
Clarity Protocol/(discovered risks),
evals/<risk>/eval_config.yaml(one atomic config per risk),agent.py, and aREADME:
azure_doc_qa,billing_support_agent,career_health_assessment,change_control_agent,prompt_agents,science_research_agent,travel_planner_langgraph,travel_planner_neurosan.the workflow, not a checked-in file.
Framework ACS CLI fixes
assert_ai/cli.py:assert-ai acs …now loads.envand refreshes Azure auth mode, soacs generateresolves credentials the same wayassert-ai rundoes. The ACS subcommandsdon't import the runner, so the project
.envwas previously never loaded.assert_ai/cli.py:acs validatenow explains that annotator-conditioned policies(
input.annotations.*) cannot fire in offline validation, so an unblocked/0-handled resultis expected rather than a policy defect.
assert_ai/integrations/acs/{prompt_builder,validate}.py: policy generation/validation fixes,with tests in
tests/test_acs_{prompt_builder,validate}.py.Docs
README.md: new skill-first Get started covering onboard → explore → use → ACS. Also repointsa quick-install command that broke when example configs moved under
evals/<risk>/.instead of the raw
policy_violation/overrefusalkeysTesting
assert-ai run→ judged artifacts underartifacts/results/). Testing matrix found here: https://onedrive.cloud.microsoft/:b:/a@tm94la6n/r/_layouts/15/onedrive.aspx?id=%2Fa%40tm94la6n%2FDocuments%2FClarityAssertAcsResultsV0%2E2Release%2FClarityAssertAcsV0%2E2Metrics%2Epdf&parent=%2Fa%40tm94la6n%2FDocuments%2FClarityAssertAcsResultsV0%2E2Release&share=cQrJBugNB1yZTrZSjrMxSfPfEgUCaq4K6MgvoSpLTJpXVJACgQChecklist
pytestand/or viewer checks as applicable).CHANGELOG.mdentry is included.