Skip to content

redteam suite: non-blocking follow-ups from #29 review #37

Description

@stefanoferi

Non-blocking follow-ups from the review of #29 (admina-redteam detection-efficacy suite). These do not block the merge of #29; tracking them here so they are not lost.

  • Corpus size / reporting — corpora are small (37/26/11 positives) and per-lang/per-tag cells are 1–3 samples. Show numerator/denominator per cell and mark the output indicative; grow the corpora (especially non-EN) over time.
  • Label provenance — labels are author-assigned in-repo. Add a one-line note that they are not third-party adjudicated.
  • CLI not shippedscripts/redteam.py is not in [project.scripts], so a pip-installed user cannot run the CLI the docs reference. Add an entry point (e.g. admina redteam) or adjust the docs.
  • Gate fp invariant — the gate compares raw fp counts, sound only because the corpora are sealed. Add assert cur["fp_samples"] == base["fp_samples"], or compare rates.
  • Corpus edge casesloop-en-009 (counter-reset labelled loop), non-Luhn test cards, and no hard negatives in the PII FPR set.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions