fix(ci): require the Codex credential before CI-investigator activation

[groupthinking]

Parent program: #898
Canonical implementation: #948

Scope

• Make the EventRelay CI Investigator fail before activation when the dedicated CODEX_API_KEY secret is absent.
• Preserve secret redaction and do not add, print, or mutate credential values.
• Keep the .md source, generated .lock.yml, audit catalog, and governance regression synchronized.

Acceptance

• The dedicated check executes before the compiled activation fallback validator.
• Tests inspect compiled behavior, not source text alone.
• gh-aw source and generated lock are synchronized.
• Exact-head CI, Coverage, gh-aw Validation, CodeQL, Security Scan, Secret Scan, and Dependency Review pass.
• All review findings proven fixed are resolved; zero threads remain unresolved.
• A current-head independent reviewer attests to 5ab39e2e9d8567233bac388964080dff50edb300.
• PR remains draft until the protected secret is configured and a real investigator run proves activation.

Exact-head evidence

• Head: 5ab39e2e9d8567233bac388964080dff50edb300
• Pinned compile: gh-aw v0.82.14, 1 workflow, 0 errors, 0 warnings.
• Focused governance tests: 9 passed.
• CI run 29950283485: passed.
• Coverage run 29950282778: passed, including test execution and artifact upload.
• gh-aw Validation 29950283640: passed.
• CodeQL 29950283484, Security Scan 29950283409, Secret Scan 29950282993, Dependency Review 29950283472: passed.
• Review threads: 0 unresolved after exact-head verification.
• E2E: skipped; this workflow-only change does not claim deployment proof.

Execution receipt

Original external execution:

• Agent login: Copilot
• Run ID: not supplied by provider; no run receipt is claimed.
• Branch / PR: copilot/fix-key-unset-codex / fix(ci): require Codex credential before investigator activation #948
• Claimed: 2026-07-22T18:28:10Z
• Latest observed heartbeat: 2026-07-22T19:21:00Z
• Exact head: 5ab39e2e9d8567233bac388964080dff50edb300

Direct controller remediation:

• Reused canonical PR fix(ci): require Codex credential before investigator activation #948; no competing branch or PR created.
• Moved the gate to compiled pre-activation, regenerated the lock, added compiled-order regression coverage, and resolved only the two proven-fixed threads.
• No delegated run ID or artifact-ready event was fabricated.

Protected next action

A repository owner must configure/authorize the protected CODEX_API_KEY, then run the investigator at this exact head. Do not post the secret.

[linear-code]

GRV-104

[github-actions]

👋 Thanks for opening this issue! It has been automatically labeled for triage.

[coderabbitai]

🔗 Related PRs

#277 - Vercel remediation complete: waitUntil adoption + active middleware + @vercel/functions package (resolves three user-defined outcomes; 16-agent verification-gate consolidated PASS on full matrix) [closed]
#404 - fix(web): sync package-lock with @vercel/functions (unblocks #277 auto-merge) [merged]
#522 - fix(tests): remove hardcoded Gemini API key from test_gemini_api.sh (#495 gap) [merged]
#872 - ci: enforce deterministic agent completion truth gate [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Summary

• Add the CODEX_API_KEY guard via the on.steps frontmatter hook (not pre-agent-steps) so it compiles into the pre_activation job and runs before the lenient CODEX_API_KEY || OPENAI_API_KEY fallback validator in activation — this is the whole point of the ticket and the fix the #948 review demanded.
• Recompile with pinned gh-aw v0.82.14 so .md and .lock.yml stay in lockstep and pass the gh-aw-validation.yml diff gate.
• Lock the behavior with a new compiled-behavior regression test that asserts the guard lives in pre_activation, activation depends on pre_activation, the fallback validator survives, and the guard never leaks into the agent job.
• Sync the README/AUDIT catalog to document the new dedicated-credential requirement, keeping the secret redacted throughout.

Design Choices

Design Choice 1: Which frontmatter hook injects the dedicated CODEX_API_KEY guard so it runs before the fallback validator?

Options Considered:

1. pre-agent-steps block (mirrors focused-coverage-controller.md) — compiles into the agent job, i.e. too late; fails the ordering requirement.
2. on.steps: block in frontmatter — per gh-aw v0.82 docs, on.steps injects deterministic steps into the pre_activation job, which runs before activation; the guard lands upstream of the fallback validator as step require_codex_credential.

Chosen Option: 2

Rationale: This is the crux of the ticket and where a naive reading of the existing codebase goes wrong. focused-coverage-controller.md uses pre-agent-steps, so it's tempting to copy that — but that hook compiles into the agent job, which runs after activation. That means when both CODEX_API_KEY and OPENAI_API_KEY are missing, the compiler's validate-secret fallback in activation fails first and the dedicated guard never executes — directly violating the "executes before the compiled activation fallback validator" acceptance criterion. This is the exact regression the canonical PR #948 review caught. on.steps is the only hook that places the guard in pre_activation ahead of the activation fallback validator, satisfying the ordering criterion and matching the canonical #948 implementation.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Enforce the credential gate at pre-activation and regenerate the lock

This phase adds the dedicated, single-secret guard to the workflow source at the earliest possible gate and regenerates the compiled artifact so the source/lock pair stays synchronized. The intent is a fast, cheap, deterministic failure with a clear operator-facing error before any AI execution or the lenient dual-secret fallback runs.

Task 1: Add the dedicated CODEX_API_KEY guard to the workflow source

Add a pre-activation credential gate to the EventRelay CI Investigator frontmatter.

• Update .github/workflows/eventrelay-ci-investigator.md frontmatter to add a steps: block under on: (the on.steps hook), with a single step named "Require dedicated Codex credential" and step id: require_codex_credential.
• The step reads CODEX_API_KEY via env: { CODEX_API_KEY: ${{ secrets.CODEX_API_KEY }} } and runs a guard that exits non-zero with an ::error:: annotation (e.g. "Dedicated CODEX_API_KEY is required") when the value is empty.
• Critical constraint: the guard must only test emptiness ([ -z "${CODEX_API_KEY}" ]) — never echo, log, export, or otherwise print/mutate the secret value, preserving redaction. Do not reference OPENAI_API_KEY in this guard (single-secret requirement, no fallback).
• Do not touch the existing permissions, engine: codex, model, network, or safe-outputs frontmatter.

Task 2: Recompile and verify source/lock synchronization

Regenerate the compiled workflow so pre_activation carries the guard ahead of the fallback validator.

• Recompile using the pinned runtime: gh aw compile eventrelay-ci-investigator --validate --approve with gh-aw v0.82.14 (the version pinned by gh-aw-validation.yml and .github/aw/actions-lock.json).
• Verify the regenerated .github/workflows/eventrelay-ci-investigator.lock.yml places the require_codex_credential step inside the pre_activation job, that activation still declares needs: pre_activation, and that the compiler-generated Validate CODEX_API_KEY or OPENAI_API_KEY secret (validate-secret) step remains in activation as the downstream fallback.
• Confirm no unintended diff in the other two lock files, since gh-aw-validation.yml diff-checks all three; only eventrelay-ci-investigator.lock.yml should change.

🤖 Prompt for AI agents

Implement Phase 1: give the EventRelay CI Investigator workflow a hard fail-fast
credential gate that runs before the compiler's lenient fallback validator, then
regenerate the compiled lock file so source and lock stay in sync.

- In `.github/workflows/eventrelay-ci-investigator.md`, add a `steps:` block
under the `on:` frontmatter key (the `on.steps` hook — do NOT use
`pre-agent-steps`, since that compiles into the `agent` job and runs too late).
- Define exactly one step named "Require dedicated Codex credential" with `id:
require_codex_credential`.
- Set `env: { CODEX_API_KEY: ${{ secrets.CODEX_API_KEY }} }` on the step, and
write a guard script that checks only for emptiness (e.g. `[ -z
"${CODEX_API_KEY}" ]`) and exits non-zero with an `::error::` annotation such as
"Dedicated CODEX_API_KEY is required" when empty.
- Hard constraint: never echo, log, export, or otherwise print/mutate the secret
value — emptiness-check only, preserving GitHub's secret redaction. Do not
reference or fall back to `OPENAI_API_KEY` in this guard.
- Leave all other frontmatter (`permissions`, `engine: codex`, `model`,
`network`, `safe-outputs`) untouched.
- Recompile with the pinned gh-aw version: run `gh aw compile
eventrelay-ci-investigator --validate --approve` using gh-aw `v0.82.14`
(matching the pin in `gh-aw-validation.yml` and `.github/aw/actions-lock.json`).
- Verify the regenerated
`.github/workflows/eventrelay-ci-investigator.lock.yml`: the
`require_codex_credential` step must be inside `jobs.pre_activation.steps`,
`jobs.activation` must still declare `needs: pre_activation`, and the
compiler-generated `validate-secret` step ("Validate CODEX_API_KEY or
OPENAI_API_KEY secret") must remain intact in `jobs.activation.steps` as the
fallback.
- Confirm the other two lock files (`canonical-pr-remediator.lock.yml`,
`focused-coverage-controller.lock.yml`) show no diff, since
`gh-aw-validation.yml` runs `git diff --exit-code` across all three.

Phase 2: Prove compiled ordering with a governance regression test

This phase locks the behavior in with a test that inspects the compiled artifact, not just source text — satisfying "tests inspect compiled behavior" and guarding against a future refactor silently moving the guard back into the agent job (where it would be too late).

Author the compiled-order regression test

Add a new governance test that asserts the guard's compiled position and the surviving fallback.

• Add test_ci_investigator_requires_dedicated_codex_credential to tests/unit/test_gh_aw_workflow_governance.py.
• Load .github/workflows/eventrelay-ci-investigator.lock.yml as plain YAML (reuse the existing _load_yaml helper — .lock.yml is standard YAML, not frontmatter).
• Assert compiled behavior: the require_codex_credential step exists within jobs.pre_activation.steps; jobs.activation.needs includes pre_activation; the fallback validate-secret step still exists in jobs.activation.steps; and the dedicated guard is absent from jobs.agent.steps (proving it did not regress into the agent job).
• Assert redaction/intent: the guard's env.CODEX_API_KEY == "${{ secrets.CODEX_API_KEY }}", the guard run contains the missing-credential error text, and does not reference OPENAI_API_KEY.
• Follow the existing style in the file (structural YAML assertions, no execution), mirroring test_focused_coverage_controller_can_read_authoritative_runs but targeting compiled job-graph structures rather than .md frontmatter.

🤖 Prompt for AI agents

Implement Phase 2: add a regression test that inspects the *compiled* lock file
(not the markdown source) to lock in the credential-gate ordering and guard
against future regressions.

- Add a new test function
`test_ci_investigator_requires_dedicated_codex_credential` to
`tests/unit/test_gh_aw_workflow_governance.py`.
- Load `.github/workflows/eventrelay-ci-investigator.lock.yml` as plain YAML
using the existing `_load_yaml` helper already used in this file (it's standard
YAML, not frontmatter).
- Assert compiled structure:
  - `require_codex_credential` step exists within `jobs.pre_activation.steps`.
  - `jobs.activation.needs` includes `pre_activation`.
  - The fallback `validate-secret` step still exists in `jobs.activation.steps`.
- The dedicated guard step is absent from `jobs.agent.steps` (proving it never
regressed into the agent job).
- Assert redaction and intent:
  - The guard step's `env.CODEX_API_KEY` equals `${{ secrets.CODEX_API_KEY }}`.
  - The guard's `run` script contains the missing-credential error text.
  - The guard's `run` script does not reference `OPENAI_API_KEY`.
- Match the existing test style in the file (structural YAML assertions only, no
execution), similar in spirit to
`test_focused_coverage_controller_can_read_authoritative_runs`, but this test
targets compiled job-graph structures rather than `.md` frontmatter.

Phase 3: Synchronize the audit catalog

This phase closes the "audit catalog synchronized" criterion by making the human-readable inventories reflect that the CI Investigator now requires a dedicated Codex credential before activation.

Document the credential requirement in the catalog and audit matrix

Record the new gating behavior in both governance docs.

• Update .github/workflows/README.md Workflow Catalog to add a row for eventrelay-ci-investigator.md, noting its trigger, purpose, and the new dedicated CODEX_API_KEY pre-activation gate.
• Update .github/workflows/AUDIT.md Decision Matrix with a corresponding entry describing the credential-gating decision and rationale (fail fast before activation; no OPENAI_API_KEY fallback for this dedicated guard).
• Keep wording consistent with existing table conventions; mention the secret by name only as a requirement — do not embed any secret values.

🤖 Prompt for AI agents

Implement Phase 3: sync the human-readable governance docs so they reflect the
new dedicated-credential gating behavior added in Phase 1.

- In `.github/workflows/README.md`, update the Workflow Catalog table to add or
update the row for `eventrelay-ci-investigator.md`, noting its trigger, purpose,
and the new dedicated `CODEX_API_KEY` pre-activation gate.
- In `.github/workflows/AUDIT.md`, update the Decision Matrix with a
corresponding entry describing the credential-gating decision and its rationale:
fail fast before activation, with no `OPENAI_API_KEY` fallback for this
dedicated guard.
- Match the existing table/wording conventions used elsewhere in both files.
- Do not embed any secret values — reference the secret only by name
(`CODEX_API_KEY`) as a stated requirement.

Research

This repo drives three custom gh-aw (GitHub Agentic Workflows) markdown sources under .github/workflows/ (eventrelay-ci-investigator.md, canonical-pr-remediator.md, focused-coverage-controller.md). Each .md compiles via a pinned gh aw compile (gh-aw v0.82.14) into a .lock.yml whose job graph includes pre_activation → activation → agent → detection/safe_outputs → conclusion. .github/workflows/gh-aw-validation.yml recompiles all three and runs git diff --exit-code to guarantee source/lock parity, plus actionlint/zizmor/poutine. Governance is enforced in Python via tests/unit/test_gh_aw_workflow_governance.py, and human-readable inventories live in .github/workflows/README.md (Workflow Catalog) and .github/workflows/AUDIT.md (Decision Matrix).

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: give the EventRelay CI Investigator workflow a hard fail-fast
credential gate that runs before the compiler's lenient fallback validator, then
regenerate the compiled lock file so source and lock stay in sync.

- In `.github/workflows/eventrelay-ci-investigator.md`, add a `steps:` block
under the `on:` frontmatter key (the `on.steps` hook — do NOT use
`pre-agent-steps`, since that compiles into the `agent` job and runs too late).
- Define exactly one step named "Require dedicated Codex credential" with `id:
require_codex_credential`.
- Set `env: { CODEX_API_KEY: ${{ secrets.CODEX_API_KEY }} }` on the step, and
write a guard script that checks only for emptiness (e.g. `[ -z
"${CODEX_API_KEY}" ]`) and exits non-zero with an `::error::` annotation such as
"Dedicated CODEX_API_KEY is required" when empty.
- Hard constraint: never echo, log, export, or otherwise print/mutate the secret
value — emptiness-check only, preserving GitHub's secret redaction. Do not
reference or fall back to `OPENAI_API_KEY` in this guard.
- Leave all other frontmatter (`permissions`, `engine: codex`, `model`,
`network`, `safe-outputs`) untouched.
- Recompile with the pinned gh-aw version: run `gh aw compile
eventrelay-ci-investigator --validate --approve` using gh-aw `v0.82.14`
(matching the pin in `gh-aw-validation.yml` and `.github/aw/actions-lock.json`).
- Verify the regenerated
`.github/workflows/eventrelay-ci-investigator.lock.yml`: the
`require_codex_credential` step must be inside `jobs.pre_activation.steps`,
`jobs.activation` must still declare `needs: pre_activation`, and the
compiler-generated `validate-secret` step ("Validate CODEX_API_KEY or
OPENAI_API_KEY secret") must remain intact in `jobs.activation.steps` as the
fallback.
- Confirm the other two lock files (`canonical-pr-remediator.lock.yml`,
`focused-coverage-controller.lock.yml`) show no diff, since
`gh-aw-validation.yml` runs `git diff --exit-code` across all three.
===============================================================================

Task: 2

Implement Phase 2: add a regression test that inspects the *compiled* lock file
(not the markdown source) to lock in the credential-gate ordering and guard
against future regressions.

- Add a new test function
`test_ci_investigator_requires_dedicated_codex_credential` to
`tests/unit/test_gh_aw_workflow_governance.py`.
- Load `.github/workflows/eventrelay-ci-investigator.lock.yml` as plain YAML
using the existing `_load_yaml` helper already used in this file (it's standard
YAML, not frontmatter).
- Assert compiled structure:
  - `require_codex_credential` step exists within `jobs.pre_activation.steps`.
  - `jobs.activation.needs` includes `pre_activation`.
  - The fallback `validate-secret` step still exists in `jobs.activation.steps`.
- The dedicated guard step is absent from `jobs.agent.steps` (proving it never
regressed into the agent job).
- Assert redaction and intent:
  - The guard step's `env.CODEX_API_KEY` equals `${{ secrets.CODEX_API_KEY }}`.
  - The guard's `run` script contains the missing-credential error text.
  - The guard's `run` script does not reference `OPENAI_API_KEY`.
- Match the existing test style in the file (structural YAML assertions only, no
execution), similar in spirit to
`test_focused_coverage_controller_can_read_authoritative_runs`, but this test
targets compiled job-graph structures rather than `.md` frontmatter.
===============================================================================

Task: 3

Implement Phase 3: sync the human-readable governance docs so they reflect the
new dedicated-credential gating behavior added in Phase 1.

- In `.github/workflows/README.md`, update the Workflow Catalog table to add or
update the row for `eventrelay-ci-investigator.md`, noting its trigger, purpose,
and the new dedicated `CODEX_API_KEY` pre-activation gate.
- In `.github/workflows/AUDIT.md`, update the Decision Matrix with a
corresponding entry describing the credential-gating decision and its rationale:
fail fast before activation, with no `OPENAI_API_KEY` fallback for this
dedicated guard.
- Match the existing table/wording conventions used elsewhere in both files.
- Do not embed any secret values — reference the secret only by name
(`CODEX_API_KEY`) as a stated requirement.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!