docs: CI gate entry point, acs CLI reference, and two missing target rungs - #281
docs: CI gate entry point, acs CLI reference, and two missing target rungs#281Chang Liu (changliu2) wants to merge 8 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An earlier commit on this branch claimed 'ASSERT has no native HTTP target' and
told users to hand-write a requests shim. That is wrong. target.endpoint is
documented in docs/targets/callable.md and fully implemented: config.py accepts
it, config_model.py validates exactly one of model/connector/callable/endpoint,
and security.py validates the URL. The runtime POSTs {message, history} and
reads {response}.
Publishing that claim would have sent every black-box user down a wrapper they
did not need, and contradicted our own reference docs one directory away.
target.endpoint is now the documented black-box path in the decision tree, the
prose, and the paths table. The callable shim is kept but re-scoped to its real
use: a service whose request/response shape differs from the one
target.endpoint expects. Applied identically to all three run-assert-eval
front-doors.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Every link in this PR pointed at responsibleai/assert-action, which is archived and private. Merging it would have published a 404 into the README of a public repo and handed bugbash participants a bootstrap URL that cannot be fetched. The action is live at changliu2/assert-ai-action (public, v1 tagged) until it can be transferred into the org. Verified the bootstrap URL returns HTTP 200. Re-point these to the org path once the transfer lands -- raw.githubusercontent does not reliably follow repo-transfer redirects. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
The npx path installs both skills across 40+ agents and is the action's recommended entry point, but this PR only documented the paste-a-URL fallback. A reader on Cursor, Gemini CLI, Windsurf, or Codex would have concluded we support three assistants when we support forty. Also states what the agent actually does, and that the gate blocks on evidence of harm rather than on every non-green outcome, so nobody mistakes it for a compliance gate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
…the user's repo The run-assert-eval skill and its mirrored prompt both told the agent to run `pip install -e ".[otel,langgraph]"` when `assert-ai --help` failed. That is correct only inside a clone of ASSERT itself. In a customer repo -- which is where this skill actually runs, and where the CI gate invokes it -- it installs the customer's own package and ASSERT is still missing, so the precondition it is meant to satisfy stays broken. Install from PyPI instead, note the route-specific extras, and keep the editable form documented for the one case where it is right. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
tangym
left a comment
There was a problem hiding this comment.
The ownership split looks clear: run-assert-eval owns local evaluation, wire-assert-ci generates the repository integration, and the published Action owns deterministic baseline comparison and CI enforcement. Agent-assisted setup also makes sense as the preferred path, with direct workflow authoring available for advanced users.
One concern is that run-assert-eval could drift between ASSERT core and the Action repo. Could we either npx install the two canonical skills from their respective repos, or mechanically sync the Action repo’s copy from ASSERT with a CI freshness check?
The documented one-liner was:
npx skills add changliu2/assert-ai-action --skill "*" --yes
That was correct when the action repo vendored a copy of run-assert-eval. It no
longer does -- the copy was removed because it had already drifted from the
version owned here in ASSERT, and a copy nobody diffs is a fork. run-assert-eval
is now installed from this repository instead, which is the only way it cannot
go stale.
So the one-liner now resolves a single skill. A reader following it gets
wire-assert-ci with nothing to delegate evaluation to, and the failure does not
appear until mid-run, well after setup looks like it succeeded. Verified against
a clean repo: the old command installs wire-assert-ci only; the two commands
below install both, sourced from their respective repositories.
They have to stay separate. `skills add` accepts one package per invocation and
silently ignores extras while still exiting 0, so a combined command looks like
it worked and quietly leaves the user with half the bundle. That is called out
in the docs rather than left as a trap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
The action now lives in the org, public, with v1 and its releases. These docs are the customer entry point for it, so shipping them still naming the personal repo would send every reader to the copy we intend to retire. Verified the bootstrap URL resolves at the new owner before committing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
… docs - docs/targets/README.md: the "pre-collected OTel traces" rung (decision table, offline-path section, and paths-at-a-glance table) claimed `judge-traces` "scores" a trace file. It doesn't -- it parses OTel spans into inference_set.jsonl and explicitly tells the caller to run --force-stage judge to score them. Verified against tests/fixtures/sample_otel_traces.json: output dir has inference_set.jsonl only, no scores.jsonl. All three references now state the real two-step path (judge-traces, then run --config <path> --force-stage judge). - .claude/skills/run-assert-eval/SKILL.md and .github/prompts/run-assert-eval.prompt.md: dropped the nonexistent `assert-ai[aiohttp]` extra (not in pyproject.toml, not on PyPI) in favor of stating aiohttp already ships transitively via litellm's own dependency, so target.endpoint works without a separate install step. - docs/cli/commands.md: `acs generate` now states it requires the `acs` extra via an editable repo install, since assert-ai 0.1.0 on PyPI does not yet publish that extra. - docs/README.md: registered the new docs/ci/README.md in the docs index (CI section) so it's reachable from the docs landing page, not just the top-level repo README. Verified: tests/test_framework_agnostic.py -k JudgeTraces still passes (2/2), confirming the new wording matches actual command behavior.
|
Ran an independent source-verification pass on this PR (checked claims against
Also registered the new Verified |
Summary
Adds a CI-gating entry point to the docs, documents the missing
acscommand group, and closes two gaps in the target decision tree that the CI onboarding skill depends on.Motivation
The README had no path from "my eval passes locally" to "this blocks bad PRs". Separately, two things the docs said (or omitted) were actively misleading for anyone wiring up a target.
Changes
CI entry point — a short README section pointing at the Action and the coding-agent bootstrap URL, plus a brief
docs/ci/README.mdthat links out rather than duplicating the Action's docs.docs/cli/commands.md: document theacsgroup.acs generate,acs validate, andacs eval-configwere documented only indocs/guides/securing-agents-with-acs.mdand were absent from the CLI reference. Flags were verified againstassert_ai/cli.py, not just the guide.docs/targets/README.md: two missing rungs.assert-ai judge-traces --traces <path> --config <path>. This only appeared in the CLI reference, never in the decision tree. Now stated explicitly, including that there is no--traceflag onassert-ai run— a natural assumption that is wrong.requestsshim. That was wrong:target.endpointexists and is fully implemented (config.py,config_model.py,security.py). The runtime POSTs{"message": ..., "history": [...]}and reads{"response": ...}.target.endpointis now the documented black-box path; the callable shim is re-scoped to its real use — a service whose request/response shape differs.run-assert-evalskill: CI hand-off + the corrected HTTP guidance, applied identically across all three front-doors (.claude/skills/,.github/prompts/,.cursor/rules/). Minimal additive edits — the skill's logic, structure, and voice are unchanged. It keeps owning running and reporting; thewire-assert-ciskill owns CI wiring.Where the Action lives
Links here point at
changliu2/assert-ai-action(public,v1tagged, CI green).The original
responsibleai/assert-actionwas archived on Jul 14 and is read-only. Neither I nor the repo owner has admin to unarchive it, and pushes to newly createdresponsibleai/*repos are also denied, so the Action could not be hosted in the org. It is fully transferable once someone with org rights can accept it:Bootstrap URL verified live (HTTP 200):
Discrepancy found
assert_ai/cli.pysupports--require-blockonacs generate, butdocs/guides/securing-agents-with-acs.mdomits it. The CLI reference documents the code's actual behaviour. The guide is left alone here but is now known to be incomplete.Testing
pyproject.toml, workflows, or viewer changes