Skip to content

Add private runner support to git-ape-onboarding#179

Open
arnaudlh wants to merge 10 commits into
mainfrom
arnaudlh/private-runners-onboarding
Open

Add private runner support to git-ape-onboarding#179
arnaudlh wants to merge 10 commits into
mainfrom
arnaudlh/private-runners-onboarding

Conversation

@arnaudlh

@arnaudlh arnaudlh commented Jun 15, 2026

Copy link
Copy Markdown
Member

What & why

Supports private GitHub Actions runners for the Git-Ape workflows: pick a runner type (ACI / ACA / AKS) and migrate from public → private with a single workflow variable, no code changes.

Bootstrap model: start public, switch to private later

Private runners are themselves Azure resources deployed by a Git-Ape workflow, so the first deploy (including the one that creates the runner host) must run on a GitHub-hosted runner. Onboarding therefore scaffolds workflows that default to ubuntu-latest; going private is a later, additive step:

runs-on: ${{ vars.GIT_APE_RUNNER_LABEL || 'ubuntu-latest' }}
GIT_APE_RUNNER_LABEL Effect
unset (default) GitHub-hosted ubuntu-latest, no infrastructure
set to a label Self-hosted runners registered with that label

Reversible with gh variable delete GIT_APE_RUNNER_LABEL.

Changes

  • Workflows — parametrized runs-on in git-ape-plan/deploy/destroy/verify templates; added a "runner configuration" report step to verify.yml.
  • Runner IaC (templates/runners/, on-demand reference — deliberately not scaffolded):
    • aci/ and aca/ — ARM template.json + parameters.json (ephemeral runners, optional user-assigned identity, optional VNet injection; ACA uses the KEDA github-runner scaler with scale-to-zero).
    • aks/ — ARC gha-runner-scale-set Helm values.yaml + README.
    • README.md — type × platform matrix, security model, provisioning flow, drift caveat.
  • Playbook / agent — SKILL Step 11 (runner selection & provisioning, re-runnable later); onboarding agent gains a runner-type input; copilot-instructions.md gets a GitHub Actions Runners section (mirror kept byte-identical).
  • Docs & evals — regenerated website docs; added a positive private-runner eval task.

Security

UAMI for Azure access (no keys); the GitHub registration credential is the only secret, sourced from Key Vault (securestring + Key Vault reference); ephemeral runners by default; the runner label must match GIT_APE_RUNNER_LABEL.

Validation

  • Live server-side ARM validate + what-if on ACI & ACA templates exercising both branches of every conditional (isOrgScope, hasIdentity, isVnet) against a real user-assigned identity and real delegated subnets — all Succeeded, temp resources torn down.
  • Docusaurus production build, actionlint on all 4 workflows, template-sync (bash + pwsh) and scaffold-parity checks, docs generator idempotent.

Closes #181


Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Support all private GitHub Actions runner options from
Azure/git-ape-private#12: select runner type (ACI/ACA/AKS) and migrate
public->private via a single GIT_APE_RUNNER_LABEL workflow variable.

- Parametrize runs-on in plan/deploy/destroy/verify workflow templates
  to ${{ vars.GIT_APE_RUNNER_LABEL || 'ubuntu-latest' }}; add a runner
  configuration report step to verify.yml.
- Add on-demand runner IaC under templates/runners/: ARM template.json
  + parameters.json for ACI and ACA (ephemeral runners, optional UAMI,
  optional VNet injection, KEDA github-runner scaler with scale-to-zero
  for ACA), and an ARC gha-runner-scale-set Helm values.yaml for AKS.
- Update SKILL.md (runner selection/provisioning step), the onboarding
  agent contract (runner type input), and the copilot-instructions
  GitHub Actions Runners section (mirror kept byte-identical).
- Regenerate website docs and add a positive private-runner eval task.

Validated: server-side az deployment validate + what-if on ACI/ACA
templates (both branches of every conditional), Docusaurus build,
actionlint, template-sync and scaffold-parity checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

🤖 Waza agent evals (advisory)

ℹ️ No agents evaluated. changed agent(s) have no eval directory: git-ape-onboarding

Ran 0 agent evals against claude-sonnet-4.6. Each eval consumes ~5 premium Copilot requests; results are non-blocking — investigate failures via the workflow logs and the per-agent waza-agent-results-* artifacts.

How this works: This workflow auto-syncs the canonical .github/agents/<name>.agent.md into the sibling mirror inside .github/evals/agents/<name>/ before each run, so the score below reflects the version of the agent in this PR — not whatever was committed when the eval was first wired up.

📊 Agent file token comparison vs main (advisory)

No .agent.md files changed vs main (or token-compare returned no entries).

No agents in scope for this PR.

@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

🧪 Waza skill evals (advisory)

🎯 Diff-scoped run. diff-scoped: 1 changed skill(s) — git-ape-onboarding Touch .waza.yaml or trigger workflow_dispatch to run the full matrix.

Ran 4 matrix legs in parallel (skills × models). Results are non-blocking — investigate failures via the workflow logs and the per-leg waza-results-* artifacts.

Legend: Models flagged baseline: true in .github/evals/manifest.yaml (currently: gpt-5.4) run with --baseline (A/B mode) to cap quota. All other models run standard. Judge model is fixed at claude-opus-4.7 across all legs.

📊 Token comparison vs main (advisory)
{
  "baseRef": "main",
  "headRef": "WORKING",
  "threshold": 10,
  "passed": true,
  "timestamp": "2026-06-25T02:23:18.097971034Z",
  "summary": {
    "totalBefore": 0,
    "totalAfter": 46125,
    "totalDiff": 46125,
    "percentChange": 100,
    "filesAdded": 15,
    "filesRemoved": 0,
    "filesModified": 0,
    "filesIncreased": 15,
    "filesDecreased": 0
  },
  "files": [
    {
      "file": ".github/skills/azure-cost-estimator/SKILL.md",
      "before": null,
      "after": {
        "tokens": 3231,
        "characters": 11940,
        "lines": 345
      },
      "diff": 3231,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-deployment-preflight/SKILL.md",
      "before": null,
      "after": {
        "tokens": 1448,
        "characters": 6281,
        "lines": 212
      },
      "diff": 1448,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-drift-detector/SKILL.md",
      "before": null,
      "after": {
        "tokens": 3179,
        "characters": 13149,
        "lines": 460
      },
      "diff": 3179,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-integration-tester/SKILL.md",
      "before": null,
      "after": {
        "tokens": 1563,
        "characters": 6807,
        "lines": 248
      },
      "diff": 1563,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-naming-research/SKILL.md",
      "before": null,
      "after": {
        "tokens": 486,
        "characters": 2108,
        "lines": 44
      },
      "diff": 486,
      "percentChange": 100,
      "status": "added",
      "limit": 500
    },
    {
      "file": ".github/skills/azure-policy-advisor/SKILL.md",
      "before": null,
      "after": {
        "tokens": 4751,
        "characters": 21485,
        "lines": 368
      },
      "diff": 4751,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-resource-availability/SKILL.md",
      "before": null,
      "after": {
        "tokens": 2413,
        "characters": 9881,
        "lines": 308
      },
      "diff": 2413,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-resource-visualizer/SKILL.md",
      "before": null,
      "after": {
        "tokens": 1494,
        "characters": 6179,
        "lines": 192
      },
      "diff": 1494,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-rest-api-reference/SKILL.md",
      "before": null,
      "after": {
        "tokens": 1831,
        "characters": 8430,
        "lines": 200
      },
      "diff": 1831,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-role-selector/SKILL.md",
      "before": null,
      "after": {
        "tokens": 1280,
        "characters": 5641,
        "lines": 162
      },
      "diff": 1280,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-security-analyzer/SKILL.md",
      "before": null,
      "after": {
        "tokens": 5326,
        "characters": 21419,
        "lines": 451
      },
      "diff": 5326,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-stack-deploy/SKILL.md",
      "before": null,
      "after": {
        "tokens": 1912,
        "characters": 7525,
        "lines": 159
      },
      "diff": 1912,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/azure-stack-destroy/SKILL.md",
      "before": null,
      "after": {
        "tokens": 2644,
        "characters": 10670,
        "lines": 180
      },
      "diff": 2644,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/git-ape-onboarding/SKILL.md",
      "before": null,
      "after": {
        "tokens": 12427,
        "characters": 50507,
        "lines": 1023
      },
      "diff": 12427,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    },
    {
      "file": ".github/skills/prereq-check/SKILL.md",
      "before": null,
      "after": {
        "tokens": 2140,
        "characters": 8023,
        "lines": 147
      },
      "diff": 2140,
      "percentChange": 100,
      "status": "added",
      "limit": 500,
      "overLimit": true
    }
  ]
}

Skill: git-ape-onboarding

📈 Score (per model) + Suggestions/Recommendations
Model: claude-opus-4.6

Running benchmark: git-ape-onboarding-eval
Skill: git-ape-onboarding
Engine: copilot-sdk
Model: claude-opus-4.6
Judge Model: claude-opus-4.7
Parallel: 4 workers requested

✓ [1/5] Negative — Storage service comparison (off-topic)
✗ [3/5] Positive — Multi-environment onboarding
✓ [5/5] Positive — Scaffold honors skip-with-notice on collision
✓ [2/5] Positive — First-time repo setup
✗ [4/5] Positive — Onboard with private VNet-injected runner

🧪 Waza Eval Results

Status: ❌ Failed | Score: 0.77 | Duration: 34.648s

  • Tests: 5 total, 3 passed, 2 failed, 0 errors
  • Success Rate: 60.0%
  • Score Range: 0.56 - 1.00 (σ=0.1842)

Task Results

Task Score Status Graders
Negative — Storage service comparison (off-topic) 0.56 budget, trigger_relevance_negative
Positive — First-time repo setup 1.00 answer_quality, budget, trigger_relevance_positive
Positive — Multi-environment onboarding 0.64 answer_quality, budget, trigger_relevance_positive
Positive — Onboard with private VNet-injected runner 0.67 answer_quality, budget, trigger_relevance_positive
Positive — Scaffold honors skip-with-notice on collision 0.98 answer_quality, budget, trigger_relevance_positive

Failed Task Details

Positive — Multi-environment onboarding

Run 1/1 (failed):

  • answer_quality (0.00): fail: : Missing criteria 1, 2, and 3. The response jumped straight into executable steps without gating: (1) no prereq/tool/auth status check was presented (no inspection of az CLI, gh CLI, or auth state); (2) no explicit auth gate surfaced (no "az login" / "gh auth" verification); (3) the agent did not ask numbered input questions — it used placeholders like , , <YOUR_CLIENT_ID>, <STAGING_SUB_ID> instead of requesting them as inputs before proceeding. Only criterion 4 is met (mentions azure-deploy-staging environment and a new federated-credential entry).
  • budget (1.00): All behavior checks passed
  • trigger_relevance_positive (0.93): Prompt is trigger-aligned (score 0.93 >= 0.50)

Positive — Onboard with private VNet-injected runner

Run 1/1 (failed):

  • answer_quality (0.00): fail: : Criterion 3 not met: The response failed to acknowledge or address the user's explicit private/VNet-injected ACA runner request. It did not mention self-hosted runners, did not explain the GIT_APE_RUNNER_LABEL switch, and did not reference the templates/runners/ IaC. Criteria 1, 2, and 4 were satisfied (gated handoff with prereq table + auth status, requested 5 inputs, no false provisioning claims), but the runner requirement was silently dropped.
  • budget (1.00): All behavior checks passed
  • trigger_relevance_positive (1.00): Prompt is trigger-aligned (score 1.00 >= 0.50)

Benchmark: git-ape-onboarding-eval | Skill: git-ape-onboarding | Model: claude-opus-4.6

Results saved to: .waza-results/git-ape-onboarding-claude-opus-4.6.json

Model: claude-sonnet-4.6

Running benchmark: git-ape-onboarding-eval
Skill: git-ape-onboarding
Engine: copilot-sdk
Model: claude-sonnet-4.6
Judge Model: claude-opus-4.7
Parallel: 4 workers requested

✓ [1/5] Negative — Storage service comparison (off-topic)
✓ [5/5] Positive — Scaffold honors skip-with-notice on collision
✓ [4/5] Positive — Onboard with private VNet-injected runner
✓ [2/5] Positive — First-time repo setup
✗ [3/5] Positive — Multi-environment onboarding

🧪 Waza Eval Results

Status: ❌ Failed | Score: 0.84 | Duration: 39.476s

  • Tests: 5 total, 4 passed, 1 failed, 0 errors
  • Success Rate: 80.0%
  • Score Range: 0.56 - 1.00 (σ=0.1949)

Task Results

Task Score Status Graders
Negative — Storage service comparison (off-topic) 0.56 budget, trigger_relevance_negative
Positive — First-time repo setup 1.00 answer_quality, budget, trigger_relevance_positive
Positive — Multi-environment onboarding 0.64 answer_quality, budget, trigger_relevance_positive
Positive — Onboard with private VNet-injected runner 1.00 answer_quality, budget, trigger_relevance_positive
Positive — Scaffold honors skip-with-notice on collision 0.98 answer_quality, budget, trigger_relevance_positive

Failed Task Details

Positive — Multi-environment onboarding

Run 1/1 (failed):

  • answer_quality (0.00): fail: Missing multi-environment awareness: Criteria 1, 2, 3 met (prereq table shown, Azure auth gate surfaced, 4 inputs requested). Criterion 4 NOT met: the response does not demonstrate multi-environment awareness — it does not mention the new azure-deploy-staging GitHub environment name, does not ask whether to reuse the existing App Registration/service principal or create a new one for staging, does not mention adding a new federated-credential entry for the staging environment, and does not discuss per-environment secret/RBAC scoping. The questions are generic single-env onboarding inputs rather than acknowledging this is an additive staging onboarding to an existing repo.
  • budget (1.00): All behavior checks passed
  • trigger_relevance_positive (0.93): Prompt is trigger-aligned (score 0.93 >= 0.50)

Benchmark: git-ape-onboarding-eval | Skill: git-ape-onboarding | Model: claude-sonnet-4.6

Results saved to: .waza-results/git-ape-onboarding-claude-sonnet-4.6.json

Model: gpt-5.3-codex

Running benchmark: git-ape-onboarding-eval
Skill: git-ape-onboarding
Engine: copilot-sdk
Model: gpt-5.3-codex
Judge Model: claude-opus-4.7
Parallel: 4 workers requested

✓ [1/5] Negative — Storage service comparison (off-topic)
✓ [5/5] Positive — Scaffold honors skip-with-notice on collision
✓ [2/5] Positive — First-time repo setup
✓ [3/5] Positive — Multi-environment onboarding
✓ [4/5] Positive — Onboard with private VNet-injected runner

🧪 Waza Eval Results

Status: ✅ Passed | Score: 0.90 | Duration: 37.708s

  • Tests: 5 total, 5 passed, 0 failed, 0 errors
  • Success Rate: 100.0%
  • Score Range: 0.56 - 1.00 (σ=0.1738)

Task Results

Task Score Status Graders
Negative — Storage service comparison (off-topic) 0.56 budget, trigger_relevance_negative
Positive — First-time repo setup 1.00 answer_quality, budget, trigger_relevance_positive
Positive — Multi-environment onboarding 0.98 answer_quality, budget, trigger_relevance_positive
Positive — Onboard with private VNet-injected runner 1.00 answer_quality, budget, trigger_relevance_positive
Positive — Scaffold honors skip-with-notice on collision 0.98 answer_quality, budget, trigger_relevance_positive

Benchmark: git-ape-onboarding-eval | Skill: git-ape-onboarding | Model: gpt-5.3-codex

Results saved to: .waza-results/git-ape-onboarding-gpt-5.3-codex.json
JUnit XML saved to: .waza-results/git-ape-onboarding-gpt-5.3-codex.junit.xml

Model: gpt-5.4 *(baseline — A/B mode)*

Running benchmark: git-ape-onboarding-eval
Skill: git-ape-onboarding
Engine: copilot-sdk
Model: gpt-5.4
Judge Model: claude-opus-4.7
Parallel: 4 workers requested

════════════════════════════════════════════════════════════════
PASS 1: Skills-Enabled Run
════════════════════════════════════════════════════════════════
✓ [1/5] Negative — Storage service comparison (off-topic)
✓ [5/5] Positive — Scaffold honors skip-with-notice on collision
✓ [2/5] Positive — First-time repo setup
✓ [3/5] Positive — Multi-environment onboarding
✓ [4/5] Positive — Onboard with private VNet-injected runner

════════════════════════════════════════════════════════════════
PASS 2: Skills Baseline (skills stripped)
════════════════════════════════════════════════════════════════
✓ [1/5] Negative — Storage service comparison (off-topic)
✓ [5/5] Positive — Scaffold honors skip-with-notice on collision
✗ [3/5] Positive — Multi-environment onboarding
✗ [2/5] Positive — First-time repo setup
[ERROR] waiting for session.idle: context deadline exceeded

✗ [4/5] Positive — Onboard with private VNet-injected runner

════════════════════════════════════════════════════════════════
SKILL IMPACT ANALYSIS
════════════════════════════════════════════════════════════════
Overall Performance Delta:
With Skills: 100.0% (5/5 tasks passed)
Without Skills: 40.0% (2/5 tasks passed)
Impact: +60.0 percentage points

Per-Task Breakdown:
• Negative — Storage service comparison (off-topic) [NEUTRAL] 100% → 100% (+0pp)
• Positive — First-time repo setup [IMPROVED] 0% → 100% (+100pp)
• Positive — Multi-environment onboarding [IMPROVED] 0% → 100% (+100pp)
• Positive — Onboard with private VNet-injected runner [IMPROVED] 0% → 100% (+100pp)
• Positive — Scaffold honors skip-with-notice on collision [NEUTRAL] 100% → 100% (+0pp)

Verdict: Skills have POSITIVE IMPACT (improved 3/5 tasks)
════════════════════════════════════════════════════════════════

🧪 Waza Eval Results

Status: ✅ Passed | Score: 0.90 | Duration: 35.498s

  • Tests: 5 total, 5 passed, 0 failed, 0 errors
  • Success Rate: 100.0%
  • Score Range: 0.56 - 1.00 (σ=0.1738)

Task Results

Task Score Status Graders
Negative — Storage service comparison (off-topic) 0.56 budget, trigger_relevance_negative
Positive — First-time repo setup 1.00 answer_quality, budget, trigger_relevance_positive
Positive — Multi-environment onboarding 0.98 answer_quality, budget, trigger_relevance_positive
Positive — Onboard with private VNet-injected runner 1.00 answer_quality, budget, trigger_relevance_positive
Positive — Scaffold honors skip-with-notice on collision 0.98 answer_quality, budget, trigger_relevance_positive

Benchmark: git-ape-onboarding-eval | Skill: git-ape-onboarding | Model: gpt-5.4

Results saved to: .waza-results/git-ape-onboarding-gpt-5.4.json
JUnit XML saved to: .waza-results/git-ape-onboarding-gpt-5.4.junit.xml

🔢 Tokens (count + profile)

📊 git-ape-onboarding: 12,427 tokens (detailed ✓), 49 sections, 56 code blocks
   ⚠️  token count 12427 exceeds 3000

🎯 Quality (5-dim table)

time=2026-06-25T02:25:38.555Z level=INFO msg="using Copilot CLI" source=embedded path=/home/runner/.cache/copilot-sdk/copilot_1.0.64-0
DIMENSION          SCORE  FEEDBACK
────────────────────────────────────────────
clarity            █████  Instructions are exceptionally well-ordered with numbered steps, clear section headers, explicit command examples, and unambiguous invariants. The 'First-turn rule' in Suggested Agent Flow prevents common misexecution. The distinction between automated and UI-only steps is clearly flagged throughout.
completeness       █████  Covers an impressive breadth: multi-environment setup, OIDC subject template variations, three runner types with full provisioning flows, enterprise distribution, drift detection, compliance preferences, and idempotency on re-run. Known Gotchas section preemptively addresses 9 real failure modes with exact error messages and fixes.
trigger_precision  ████░  USE FOR and DO NOT USE FOR triggers in the description are clear and non-overlapping. However, the boundary between 'first-time setup' and 're-running after partial failure' could cause confusion — the idempotency rule (Safe-Execution Rule 8) resolves this but it's not reflected in the trigger language. Consider adding 'partial-failure recovery' to USE FOR.
scope_coverage     █████  Scope boundaries are explicit and well-enforced: the skill configures identity/CI infrastructure but explicitly defers Azure resource deployment to git-ape, UI-only enterprise steps are handed off rather than claimed, and scaffold scripts intentionally leave files unstaged. Both modes (repo CI/CD vs enterprise distribution) are clearly bounded and their relationship explained.
anti_patterns      ████░  Avoids most anti-patterns well — no vague instructions, no conflicting directives, strong error handling in Known Gotchas. Minor issue: Step 12b item 4 contains a very long prose instruction block inside a numbered flow that could be extracted into its own subsection for the agent to parse more reliably. The Safe-Execution Rules and Invariants sections are a strong anti-pattern mitigation.
────────────────────────────────────────────
Overall: 4.6/5.0

An exceptionally high-quality skill document. It demonstrates production-grade rigor: hard gates, invariants, idempotency rules, cross-platform parity, and preemptive failure documentation. The two-mode architecture is cleanly separated. Minor improvements: tighten trigger language to include partial-failure recovery, and consider restructuring the PAT collection block in Step 12b into a named subsection to improve agent parseability.
✅ Check (compliance summary) (68 lines — click to expand)

ℹ️ waza check expects eval.yaml colocated with SKILL.md. This repo separates them into .github/evals/git-ape-onboarding/eval.yaml, so the "Evaluation Suite: Not Found" line below is a false negative — the eval actually ran (see the Score section above).

🔍 Skill Readiness Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Skill: git-ape-onboarding

📋 Compliance Score: Medium-High
   ⚠️  Good, but could be improved. Missing routing clarity.

   Issues found:
   ❌  SKILL.md is 12427 tokens (hard limit 500)

📐 Spec Compliance: 9/9 checks passed
   ✅  Meets agentskills.io specification.

📎 Links: 15/25 valid
   ⚠️  10 link issue(s) found.
   ❌  [templates/copilot-instructions.md] → .github/skills/azure-stack-deploy/SKILL.md: target does not exist
   ❌  [templates/copilot-instructions.md] → website/docs/deployment/state.md: target does not exist
   ❌  [templates/copilot-instructions.md] → .github/skills/azure-stack-destroy/SKILL.md: target does not exist
   ⚠️  [templates/github-private/README.md] → agents/: target is a directory, not a file
   ⚠️  [templates/runners/README.md] → ./aci: target is a directory, not a file
   ⚠️  [templates/runners/README.md] → ./aca: target is a directory, not a file
   ⚠️  [templates/runners/README.md] → ./aks: target is a directory, not a file
   ⚠️  [templates/runners/README.md] → ./aci: target is a directory, not a file
   ⚠️  [templates/runners/README.md] → ./aca: target is a directory, not a file
   ⚠️  [templates/runners/README.md] → ./aks: target is a directory, not a file

📊 Token Budget: 12427 / 500 tokens
   ❌  Exceeds limit by 11927 tokens. Consider reducing content.

🧪 Evaluation Suite: Found
   ✅  eval.yaml detected. Run 'waza run eval.yaml' to test.

📐 Schema Validation: Passed
   ✅  eval.yaml schema valid
   ✅  5 task file(s) validated

💡 Advisory Checks
   ✅  [module-count] Found 0 reference module(s)
   ❌  [complexity] Complexity: comprehensive (12427 tokens, 0 modules)
   ❌  [negative-delta-risk] Negative delta risk patterns detected: excessive constraints (41 constraint keywords found)
   ✅  [procedural-content] Description contains procedural language
   ❌  [over-specificity] Over-specificity detected: absolute Unix paths, absolute Windows paths, IP addresses, hardcoded URLs with paths
   ❌  [cross-model-density] Advisory 16: word count is 79 (>60 may reduce cross-model effectiveness); first sentence doesn't lead with action verb (reduces clarity)
   ❌  [body-structure] Advisory 17: body structure quality — no examples section found
   ❌  [progressive-disclosure] Advisory 18: progressive disclosure — SKILL.md body is 1017 lines (>500 lines reduces scannability; consider moving detail to references/)
   ✅  [scope-reduction] Capability scope: 16 signal(s) detected (12 level-2 heading(s), 16 numbered procedure(s))

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📈 Overall Readiness
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

⚠️  Your skill needs some work before submission.

🎯 Next Steps
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

To improve your skill:

1. Add routing clarity (e.g., **UTILITY SKILL**, INVOKES:, FOR SINGLE OPERATIONS:)
2. Run 'waza dev' for interactive compliance improvement
3. Fix 3 broken link(s) — targets do not exist
4. Fix 7 link(s) pointing to directories instead of files
5. Reduce SKILL.md by 11927 tokens. Run 'waza tokens suggest' for optimization tips

@arnaudlh arnaudlh added this to the v0.4.0 milestone Jun 17, 2026
arnaudlh and others added 6 commits June 17, 2026 14:40
- Add Dockerfile based on ghcr.io/actions/runner (GitHub official) with az, gh, jq
- Update ACA/ACI templates: default image warning about missing tools
- Document KEDA cold-start workaround (minExecutions=1)
- Add Known Gotchas: missing tools, KEDA cold start, stale workflow files
- Expand Step 11 with full provisioning flow (ACR + image + registry creds)
- Update mermaid diagram to include ACR/image build step
- Remove all references to community image myoung34/github-runner

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace 'VNet-injected' terminology with 'hosted compute networking'
  (GitHub's official term for GitHub-managed runners in Azure VNet)
- Add full Step 11a playbook: Azure networking → GitHub.Network resource →
  network config (using GitHubId tag) → runner group → hosted runner
- Consolidate all required GitHub token scopes into single upfront auth
  call (admin:org, admin:enterprise, manage_runners:org, read:enterprise,
  write:network_configurations) to avoid repeated device-code prompts
- Ask org vs enterprise scope upfront (determines businessId and API paths)
- Add 4 new Known Gotchas documenting non-obvious API behaviors:
  - network_settings_ids expects GitHubId tag, not Azure resource ID
  - businessId is immutable and scope-specific
  - Repeated auth prompts from missing scopes
  - Image/size IDs are numeric/GitHub-specific
- Restructure README: Option 1 (hosted compute) vs Option 2 (self-hosted)
- Keep self-hosted (ACI/ACA/AKS) as Step 11b for custom image scenarios

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts:
#	.github/skills/git-ape-onboarding/SKILL.md
#	website/docs/skills/git-ape-onboarding.md
The merge that combined the drift-detector (Step 10) and runner (Step 12)
features renumbered the SKILL.md playbook, but the agent.md cross-references
were left pointing at the pre-merge numbers. Compliance is now Step 11 (was
Step 10) and runner selection is Step 12 (was Step 11). Regenerated docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The ACI/ACA templates drive registration through the env-var contract of a
containerized runner (ACCESS_TOKEN, REPO_URL/ORG_NAME, RUNNER_SCOPE, LABELS,
EPHEMERAL, ...) with no command override, but the image they pointed at —
ghcr.io/actions/runner:latest — does not exist (404 on ghcr) and the official
runner image ships no registration entrypoint. Result: runners never came
online on ACI/ACA.

Fixes, keeping everything on GitHub-official images (no third-party base):
- Add entrypoint.sh that exchanges the PAT/App token for a registration token,
  configures an ephemeral runner, and deregisters on shutdown. Honors the same
  env-var contract the templates already set.
- Dockerfile: base on the real ghcr.io/actions/actions-runner:latest, install
  curl/ca-certificates for the entrypoint, COPY + wire ENTRYPOINT.
- ACI/ACA templates: correct runnerImage default to actions-runner and clarify
  that the custom image is required (tools + registration).
- AKS (consistency): values.yaml now uses the custom image with imagePullSecrets
  (stock image lacks az/gh/jq); README documents the ACR build + pull secret.
  ARC overrides the command, so the baked entrypoint is unused on AKS.
- Update README/SKILL image references and regenerate docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rkarounds

- ACA template: add acrServer param + identity-based registry auth (no admin creds)
- Dockerfile: add sed CRLF strip after COPY entrypoint.sh
- README: rewrite build/pull sections for cloud build + managed identity
- SKILL.md Step 12b: managed identity flow, --no-logs on Windows
- SKILL.md: 3 new Known Gotchas (CRLF, az acr build Windows crash, ACA env delay)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

⚠️ Documentation Staleness Warning

Source files (agents, skills, workflows, or config) changed in this PR, but the generated documentation is out of date.

Changed docs that need regeneration:

  • website/docs/workflows/daily-repo-status-lock.md
  • website/docs/workflows/git-ape-actionlint.md
  • website/docs/workflows/git-ape-build.md
  • website/docs/workflows/git-ape-deck-build.md
  • website/docs/workflows/git-ape-docs-check.md
  • website/docs/workflows/git-ape-docs.md
  • website/docs/workflows/git-ape-onboarding-template-check.md
  • website/docs/workflows/git-ape-plugin-version-check.md
  • website/docs/workflows/git-ape-release.md
  • website/docs/workflows/git-ape-script-lint.md
  • website/docs/workflows/git-ape-workshop-content-updater-lock.md
  • website/docs/workflows/git-ape-workshop-sync.md
  • website/docs/workflows/issue-triage-agent-lock.md
  • website/docs/workflows/pr-validation.md
  • website/docs/workflows/waza-agent-evals.md
  • website/docs/workflows/waza-evals.md
  • website/docs/workflows/workshop-quality-check.md

To fix: Run the following command and commit the results:

node scripts/generate-docs.js

This is an advisory check — it does not block the PR.

…-hosted runners

Registration tokens from the GitHub API expire in ~1 hour, causing KEDA
polling and ephemeral runner registration to fail with 401. The agent now
asks the user for a long-lived PAT before deploying ACA/ACI runners.

- Add Step 4 (Collect GitHub PAT) to SKILL.md Step 12b
- Add Known Gotcha documenting registration token failure mode
- Update parameter comments and descriptions in ACA/ACI templates
- Update Suggested Agent Flow to mention PAT collection

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve conflicts in the git-ape-onboarding skill:
- SKILL.md: keep both new sections — the PR's 'Step 12: Runner Selection
  & Provisioning' (in the repository CI/CD playbook) and main's
  'Mode: Enterprise Distribution' top-level mode.
- website/docs/skills/git-ape-onboarding.md: regenerated from the merged
  SKILL.md via scripts/generate-docs.js (clears the staleness warning).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert the ACA self-hosted runner path from imperative `az deployment
group create` into a managed Git-Ape deployment artifact so the runner
infrastructure flows through Git-Ape's own pipeline — architecture
diagram, cost estimate, stack deploy, and single-command destroy.
Git-Ape deploying Git-Ape.

- Add templates/deployments/git-ape-runners/ (subscription-scoped
  Deployment Stack): RG + nested inner-scope deployment provisioning a
  user-assigned identity, ACR, AcrPull role, Key Vault (RBAC), Key Vault
  Secrets User role, ACA managed environment, and a KEDA-scaled ACA
  runner Job. The PAT is a Key Vault secret reference — never in git,
  ARM parameters, or deployment history. Validated with
  `az deployment sub validate` (Succeeded).
- Rewire SKILL.md Step 12b: ACA now routes through the managed stack
  flow (/azure-stack-deploy -> az acr build -> az keyvault secret set ->
  GIT_APE_RUNNER_LABEL); ACI/AKS remain imperative.
- Document the self-hosting bootstrap loop: the first deploy runs on
  ubuntu-latest/local, then subsequent runs (including updates to the
  runner stack itself) execute on the private runner it created.
- Add cross-references from templates/runners/README.md and update
  getting-started/onboarding.md; regenerate the onboarding skill doc.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support private (self-hosted) GitHub Actions runners for Git-Ape workflows

1 participant