Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 2 additions & 44 deletions .github/workflows/AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ concrete reason, verified against the actual repository tree.
| `.yaml` → `stale.yml` | **FIX (rename)** | File had no basename (literally `.yaml`); renamed to `stale.yml`. Content (daily stale-bot) is sound. |
| `auto-assign.yml` | **FIX** | Replaced `gh issue edit` with the REST assignees endpoint. The CLI command used GraphQL `replaceActorsForAssignable`, which fails for this repository's GitHub App token when assigning the issue owner. |
| `auto-label.yml` | KEEP | Labels PRs by changed file type; guarded with try/catch. |
| `autonomous-video-processing.yml` | **FIX** | Was a discovery loop whose "processing" step incremented a counter and printed success, so every run reported videos as processed without doing any work. Inline heredoc extracted to `scripts/ci/autonomous_video_{plan,processing,summary}.py` (lintable + unit-tested); added `workflow_call`, secret preflight, guardrail caps, per-video correlation-ID manifests, 30-day evidence retention, and a QA-gated deliverables upload. See the "Multi-agent pipeline alignment" note below. |
| `autonomous-video-processing.yml` | KEEP | Manual matrix batch processor; well-formed, scoped permissions. |
| `branch-cleanup.yml` | **FIX** | Added `workflows: write` permission (missing permission caused push of restored branch to fail with "refusing to allow a GitHub App to create or update workflow ... without `workflows` permission"). Also restored push-sentinel trigger for `claude/branch-cleanup-*` branches and the restore-branch step, and removed the incorrect NOTE claiming restoration of workflow-containing branches is impossible with this token. |
| `bulk-issue-processor.yml` | KEEP | Manual bulk issue ops via `gh` + Python; dry-run default. |
| `ci.yml` | **FIX** | Added blocking `apps/web` type-check and ESLint steps before the build so CI fails fast on TypeScript or lint regressions. |
Expand All @@ -24,7 +24,6 @@ concrete reason, verified against the actual repository tree.
| `deploy.yml` | **DELETE** | References a non-existent `deployments/` tree (manifests/terraform); actual infra is `infrastructure/`. The validate job hard-`exit 1`s on missing manifests. Generic multi-cloud (AWS+Azure+Slack) scaffold that duplicates `deploy-cloud-run.yml`. |
| `e2e-tests.yml` | **FIX** | Resolve the PR's Vercel preview deployment via the GitHub Deployments API before E2E runs, and skip the PR-comment step for forked `pull_request` runs where `GITHUB_TOKEN` is read-only (`Resource not accessible by integration`). Same-repo PRs still get comments. |
| `emergency-stop.yml` | KEEP | Manual operational kill-switch with typed confirmation. |
| `eventrelay-ci-investigator.md` / `.lock.yml` | **FIX** | Require a dedicated `CODEX_API_KEY` credential in pre-agent steps so Codex-specific runs fail fast with an explicit key-missing error instead of ambiguous fallback behavior. |
| `issue-triage.yml` | KEEP | Keyword auto-labeling + triage comment on new issues. |
| `mcp-optimization.yml` | **DELETE** | Entire workflow targets `mcp-servers/mcp-profiling/` (requirements.txt, investigator_client.py, profiling_server.py) which does not exist — every run fails. |
| `phase-goal-tracker.yml` | KEEP | Tracks markdown checklists on phase issues, keeps a single status comment updated, and auto-closes the issue when all checklist goals are complete. |
Expand Down Expand Up @@ -72,45 +71,4 @@ The protected policy at `.github/agent-lock/trusted-publishers.json` starts with
## Repository governance workflows

| `pr-governance.yml` | **ADD** | Validates that every non-draft ready PR links exactly one real open issue (not a PR number) with non-empty delivery evidence sections (Outcome, Risk, Verification, Production evidence). Fails closed on competing implementation PRs. Triggers on `pull_request_target`. |
| `repository-reconciliation.yml` | **ADD** | Scheduled (13:17 UTC daily) non-destructive reconciliation report: identifies ready PRs missing a canonical issue, issues with competing implementation PRs (references validated via Issues API), and stale unattached branches. Excludes draft PRs and fork-branch name collisions. Upserts a single issue titled "[automation] Repository drift report". |
## Multi-agent pipeline alignment (Phase 1)

**Gate 0 decision — map, don't duplicate.** ATLAS / PRISM / FORGE / SENTINEL are
adopted as *role labels* over the pipeline stages that already exist in
`src/agents/pipeline_orchestrator.py`, not as a parallel agent system:

| Role | Existing stage |
|------|----------------|
| ATLAS | `video-ingest` |
| PRISM | `research-grounding` |
| FORGE | `code-gen` |
| SENTINEL | `quality-gate` |
| Lead Engineer | `PipelineOrchestrator` |

The mapping is a single constant (`STAGES` in
`scripts/ci/autonomous_video_processing.py`), so Phase 2 wires runners into the
existing DAG, VERA security wrapping and `PipelineAuditStore` rather than
standing up a second roster. The alternative — new modules under
`src/agents/specialized/` — was rejected: nothing in the current roster is being
retired, and duplicating it would give EventRelay two competing pipelines, which
contradicts the single-workflow principle in `CLAUDE.md` / `GEMINI.md`.

**What Phase 1 changed.** The previous workflow's processing step was
`processed += 1` under a comment reading "Real processing hook", so every run
reported success regardless of whether anything happened. Status is now derived
from actual stage records: `discovered` → `blocked`/`failed` → `delivered`, and
`delivered` requires every stage including the terminal QA stage to succeed.
While the Phase 2 runners are unregistered, `pipeline_mode: full` fails closed
with `blocked` — an honest signal — and the default `discovery` mode terminates
at `discovery-only` without ever claiming delivery.

**What Phase 1 deliberately did not do.**

- No `agents/{atlas,prism,forge,sentinel,lead_engineer}.py` — that is Phase 2 and
extends the existing `AgentRequest` / `AgentResult` DTOs in
`src/youtube_extension/services/agents/dto.py`.
- No `/master-prompt-learning/session_*.md` writer — that is Phase 3 and should
be rendered from `PipelineAuditStore` records rather than a new store.
- No `contents: write` on the workflow. Committing session records from CI needs
elevated permissions; evidence is artifact-only until that trade-off is
explicitly accepted.
| `repository-reconciliation.yml` | **ADD** | Scheduled (13:17 UTC daily) non-destructive reconciliation report: identifies ready PRs missing a canonical issue, issues with competing implementation PRs (references validated via Issues API), and stale unattached branches. Excludes draft PRs and fork-branch name collisions. Upserts a single issue titled "[automation] Repository drift report". |
49 changes: 1 addition & 48 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ workflow; this README is the index.
| Close stale issues | `stale.yml` | daily (00:00 UTC) | Mark and close stale issues and PRs |
| Branch Cleanup | `branch-cleanup.yml` | manual; push sentinel on `claude/branch-cleanup-*` | Gated archive-then-delete of branches (dry-run by default); push `[restore-branch:<branch>]` sentinel to restore a deleted branch from its archive tag |
| E2E Tests | `e2e-tests.yml` | push / PR to `main` | Run Vitest E2E pipeline tests against production or the PR's Vercel preview deployment and report results on the PR |
| Autonomous Video Processing | `autonomous-video-processing.yml` | manual; `workflow_call` | Batch-process YouTube videos by category (matrix) through the ATLAS→PRISM→FORGE→SENTINEL stage pipeline, emitting per-video correlation-ID manifests |
| Autonomous Video Processing | `autonomous-video-processing.yml` | manual | Batch-process YouTube videos by category (matrix) |
| Real Video Processing (Cloud) | `real-processing.yml` | manual | Process a single video: transcript and/or AI analysis |
| API-cost PostgreSQL | `api-cost-postgres.yml` | push / PR when substrate changes; manual | Exercise fresh, upgrade-from-002, and round-trip migrations plus runtime-role integration tests on PostgreSQL 16 |
| Deploy to Google Cloud Run | `deploy-cloud-run.yml` | manual | Run migrations, deploy the bounded delivery-disabled worker, then promote a tested API candidate |
Expand Down Expand Up @@ -73,53 +73,6 @@ Generates pytest coverage and uploads lcov to Qlty.
- The test step is authoritative (`--cov-fail-under=90`, no `continue-on-error`,
no `|| true`) so failures cannot report green.

### Autonomous Video Processing — `autonomous-video-processing.yml`

The batch video pipeline. It is the repository's first reusable workflow
(`workflow_call`), so it also establishes the convention: `workflow_dispatch`
and `workflow_call` declare the *same* input names and every step reads them
through the `inputs` context (never `github.event.inputs`), so a single job body
serves both triggers.

All logic lives in versioned, unit-tested scripts rather than inline heredocs:

| Script | Job | Responsibility |
|--------|-----|----------------|
| `scripts/ci/autonomous_video_plan.py` | `prepare` | Build the category matrix; fail closed if the batch exceeds the video or model-call cap |
| `scripts/ci/autonomous_video_processing.py` | `process` | Discover videos, run the stage pipeline, write the manifest tree |
| `scripts/ci/autonomous_video_summary.py` | `summary` | Aggregate per-category manifests into the run status and workflow outputs |

**Modes.** `pipeline_mode: discovery` (default) discovers candidates and writes
manifests without invoking any generation API — this is the dry-run path for the
whole pipeline. `pipeline_mode: full` executes every stage and fails closed while
the Phase 2 agents are unimplemented.

**Stage roles.** ATLAS, PRISM, FORGE and SENTINEL are role labels mapped onto the
existing `PipelineOrchestrator` stages (`video-ingest`, `research-grounding`,
`code-gen`, `quality-gate`) — see `STAGES` in
`scripts/ci/autonomous_video_processing.py`. They are deliberately *not* a second
agent system.

**Evidence.** Each run writes a manifest tree retained for 30 days:

```
pipeline_output/<category>/run.json
pipeline_output/<category>/videos/<video_id>/manifest.json
pipeline_output/<category>/videos/<video_id>/stages/{atlas,prism,forge,sentinel}.json
```

Every video carries a deterministic correlation ID that is repeated in each stage
record, so any artifact can be linked back to its originating run.

**Guardrails.**

- `max_videos_per_run` and `max_model_calls` are enforced in `prepare`, before any
external call; an over-budget batch never starts.
- Discovery returning zero videos is a failure, not an empty success.
- A video is `delivered` only when every stage — including the terminal SENTINEL
QA stage — reports success. The deliverables artifact upload is conditioned on
that status, so a blocked run publishes evidence but never deliverables.

### Deploy to Google Cloud Run — `deploy-cloud-run.yml`

The only backend deployment path. It remains manual (`workflow_dispatch`) so a
Expand Down
Loading
Loading