Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
- run: ruff format --check src tests tools
- run: ruff check src tests tools
- run: mypy src
- run: pytest --cov-report=xml
- run: pytest --cov=agentguard --cov-report=xml --cov-report=term-missing --cov-fail-under=85
# Detection quality is a reviewable number, not a claim. Printed on every run so a
# precision or recall change shows up in the diff of two CI logs.
- run: python -m tools.bench
Expand Down
43 changes: 41 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,43 @@ permissions:
contents: read

jobs:
# Nothing left this machine before this job existed. `git tag` on a broken commit
# published a broken package, and a PyPI version can be yanked but never replaced - so
# the recovery from a bad release is a new version number and a permanent gap. The
# release proves itself here or not at all.
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: python -m pip install -e '.[dev]'

# The tag is the only thing a human types in this pipeline, and the version is
# written in exactly one place. If they disagree, PyPI would receive a package
# declaring a version nobody tagged, permanently.
- name: Tag must match the declared version
run: |
tag="${GITHUB_REF_NAME#v}"
declared="$(python -c 'import tomllib,pathlib; print(tomllib.loads(pathlib.Path("pyproject.toml").read_text())["project"]["version"])')"
installed="$(python -c 'import importlib.metadata as m; print(m.version("agentguard-sast"))')"
echo "tag=$tag pyproject=$declared installed=$installed"
if [ "$tag" != "$declared" ] || [ "$tag" != "$installed" ]; then
echo "::error::tag '$tag' disagrees with pyproject ('$declared') / installed ('$installed')"
exit 1
fi

- run: ruff format --check src tests tools
- run: ruff check src tests tools
- run: mypy src
- run: pytest --cov=agentguard --cov-fail-under=85
- run: python -m tools.bench
- run: python -m tools.bench --field-only
- run: python -m tools.measure_linearity --check

build:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -17,6 +53,9 @@ jobs:
python-version: "3.12"
- run: python -m pip install build
- run: python -m build
# Paired with download-artifact@v8 below. The majors are not parallel: v8 is the
# documented counterpart to upload-artifact@v7, per download-artifact's release
# notes ("v8 supports downloading artifacts uploaded with actions/upload-artifact@v7").
- uses: actions/upload-artifact@v7
with:
name: distributions
Expand All @@ -31,7 +70,7 @@ jobs:
permissions:
id-token: write
steps:
- uses: actions/download-artifact@v4
- uses: actions/download-artifact@v8
with:
name: distributions
path: dist/
Expand All @@ -43,7 +82,7 @@ jobs:
permissions:
contents: write
steps:
- uses: actions/download-artifact@v4
- uses: actions/download-artifact@v8
with:
name: distributions
path: dist/
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
__pycache__/
*.py[cod]
*.egg-info/
# Root-anchored on purpose. A bare `.env` would also ignore the corpus fixtures at
# tests/corpus/**, which must stay tracked - they are the test data for .env scanning.
/.env
/.env.*
!/.env.example

.venv/
.venv*/
venv/
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type:
mypy src

test:
pytest
pytest --cov=agentguard --cov-report=term-missing --cov-fail-under=85

bench:
python -m tools.bench
Expand Down
14 changes: 9 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,16 @@ Measured against a labelled corpus in this repository. Reproduce it with `make b
| **all** | 14 | 1 | 0 | 93.3% | 100.0% |
```

**93.3% precision and 100% recall over 34 labelled files** — 13 true positives and 21 true
negatives, each carrying a written reason for its label in `tests/corpus/manifest.yml`.
That is the only accuracy figure this project publishes, because it is the only one it can
reproduce.
**93.3% precision and 100% recall over 34 labelled files** — 14 that must produce a
finding and 20 that must not, each carrying a written reason for its label in
`tests/corpus/manifest.yml`. That is the only accuracy figure this project publishes,
because it is the only one it can reproduce.

13 of the 21 true negatives reproduce a false positive observed on a real project; the other
The table counts *findings*, not files: 14 true positives from the 14 positive files, and
the single false positive is one finding on one negative file. They are one-to-one here
because no corpus case expects more than one rule to fire.
Comment on lines +40 to +42

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Describe the table as rule-file outcomes

The benchmark does not count findings: measure() groups findings by rule ID and increments each tally only once per rule per file, regardless of how many locations fired. For example, hardcoded_secrets.py can emit four AG001 findings and command_execution.py can emit three AG002 findings while each contributes only one TP, so the newly added claim that the table counts findings materially misstates both its unit and the published precision; describe these as rule-file outcomes or change the manifest and tallying to represent multiplicity.

Useful? React with 👍 / 👎.


12 of the 20 negative cases reproduce a false positive observed on a real project; the other
8 were composed to cover awkward cases — a credential in a docstring, `eval` on a literal,
`subprocess` with a fixed argument vector, a `.env` full of shell interpolation. Every case
declares which it is, and `make bench` scores the field-derived subset separately:
Expand Down
Loading
Loading