Skip to content

standardize-repo: verify GitHub-side prerequisites, empirically — four shipped automations were inert #428

Description

@evanharmon1

Invariant

A repo can be perfectly standardized on disk and still have every shipped
automation inert, because each one depends on GitHub-side state the repository
cannot see. standardize-repo verifies the files it wrote; nothing verifies that
the capabilities those files assume are actually switched on. A "standardized"
report should not be compatible with "none of the automation works".

Current violation (observed 2026-08-11)

One standardize run across ponderousdev/omator and
ponderousdev/lawnomator-site (harmon-init v4.1.1 → v4.28.0) surfaced four
independent instances of the same shape — in-repo config correct, capability off
at the GitHub layer, no signal anywhere:

Repo config GitHub reality
renovate.json has vulnerabilityAlerts.enabled: true Dependabot alerts disabled on omator → the setting produced nothing
snyk_scan_schedule: weekly, workflow rendered SNYK_TOKEN absent on both repos → the scan had been failing weekly
ponderousdev/foreman had CI_APP_CLIENT_ID set 2026-07-18 CI_APP_PRIVATE_KEY never set → ~3 weeks of workflows clearing the vars.CI_APP_CLIENT_ID != '' gate and then failing at token minting
omator ships three ruleset JSONs describing required checks zero rulesets and no branch protection active

None of these is visible from the repository. Each reads as correctly configured
in-repo, and each is silently dead.

assets/verify-applied.sh and the repo's own gate both pass in every one of
these states, by design — they check the artifact, not the environment it
assumes.

Why this belongs in the skill rather than the template

These are properties of a repository's GitHub configuration, not of generated
files, so there is nothing for the template to render. The skill is already the
component that reaches GitHub during an update (it reads release records,
capability status, project fields and labels), and it is already the component
that produces the "here is what standardizing found" report. The check belongs
next to that report.

The verification must be empirical

This is the part most likely to be got wrong, and it was got wrong during the
run that produced this issue.

gh variable list --repo <r> lists repo-scoped variables only. It does not
show organization variables the repo inherits. Reading 0 from it and concluding
"no credential" is invalid — and led directly to six unnecessary repo-level
secret writes, plus a misdiagnosis of why a workflow was not running (the real
reason: the workflow was not on main yet). The org-level endpoints return 403
without admin:org, so the absence of read access is itself an unknown, not a
negative.

What was actually conclusive: a workflow run in the repo's history that mints the
App token and succeeded. Prefer that class of evidence — did the thing that
depends on the capability actually work — over listing config that may be
inherited from a scope the caller cannot read.

The same applies to GET /repos/{owner}/{repo}/vulnerability-alerts: 204 means
enabled, but 404 means either disabled or the token lacks
Administration-read. Report unknown as unknown.

Presence checks must never print or expand a credential's value — see
evanharmon1/harmon-init#567 for the construct-level rule.

Verify

# Each of the four, read-only, on a repo the skill has just standardized:
gh api "repos/<owner>/<repo>/vulnerability-alerts" -i | head -1        # 204 on / 404 off-or-unauthorized
gh api "repos/<owner>/<repo>/rulesets" --jq 'length'                   # 0 = nothing enforced
gh secret list --repo <owner>/<repo> | grep -c SNYK_TOKEN              # repo scope only
gh run list --repo <owner>/<repo> --workflow <token-minting-wf> --limit 5 \
  --json conclusion --jq '[.[].conclusion]'                            # the empirical check

Expected on an unprepared repo: a rendered snyk-scheduled.yml with no
SNYK_TOKEN, vulnerabilityAlerts.enabled: true with a 404, and checked-in
ruleset JSON with rulesets → 0.

Acceptance criteria

  • After applying a mode, the skill reports, per shipped automation, whether
    its GitHub-side prerequisite is satisfied, unsatisfied, or unknown.
  • At minimum: Dependabot alert feed, CI_APP_CLIENT_ID / CI_APP_PRIVATE_KEY,
    SNYK_TOKEN when snyk_scan_schedule != off, and active rulesets when the
    repo ships ruleset JSON.
  • Inherited org-level configuration is never reported as absent from a
    repo-scoped listing; an unreadable scope reports unknown.
  • Where a workflow-run history can settle the question, that evidence is
    preferred over config listing.
  • The check is read-only and reports presence without printing values
    (conventions: never expand a credential env var with ${VAR:-…} to report presence harmon-init#567).
  • Unsatisfied prerequisites are reported as maintainer actions with the exact
    command or settings path — never remediated by the skill, which must not
    create credentials or change security-relevant settings.

Found during a v4.28.0 standardize run across ponderousdev/omator and
ponderousdev/lawnomator-site.

Metadata

Metadata

Assignees

No one assigned

    Labels

    domain:platformCI, build, test infra, and tooling in this repoenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions