-
Notifications
You must be signed in to change notification settings - Fork 0
chore(claude): add project settings with post-edit test hook #153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
koydas
wants to merge
3
commits into
main
Choose a base branch
from
claude/claude-codex-comparison-5kz7e9
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "permissions": { | ||
| "allow": [ | ||
| "Bash(node --test *)", | ||
| "Bash(node --test scripts/tests/*.test.mjs)" | ||
| ] | ||
| }, | ||
| "hooks": { | ||
| "PostToolUse": [ | ||
| { | ||
| "matcher": "Write|Edit", | ||
| "hooks": [ | ||
| { | ||
| "type": "command", | ||
| "command": "f=$(jq -r '.tool_input.file_path // \"\"'); if echo \"$f\" | grep -qE '(scripts/|\\.github/workflows/)'; then node --test scripts/tests/*.test.mjs; fi", | ||
| "timeout": 60, | ||
| "statusMessage": "Running test suite..." | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| name: Test | ||
| on: [push] | ||
| jobs: | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v2 | ||
| - name: Run tests | ||
| run: node --test scripts/tests/*.test.mjs | ||
| - name: Check coverage | ||
| run: | | ||
| coverage=$(jest --coverage) | ||
| if [[ $coverage =~ "Statements.*:.*80%" ]]; then | ||
| echo "Test coverage meets requirements" | ||
| else | ||
| echo "Test coverage does not meet requirements" | ||
| exit 1 | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,248 +1,7 @@ | ||
| # Code Generation MVP Setup | ||
|
|
||
| This repository includes an MVP workflow that converts validated issues into AI-generated draft pull requests. The default AI provider is **Groq** with stage-specific defaults: `validation`/`review` use `qwen/qwen3-32b`, while `generation`/`autofix` use `llama-3.3-70b-versatile`. Anthropic (Claude models) is also supported and can be selected via the `AI_PROVIDER` environment variable when both provider keys are configured. The workflow triggers automatically when the validation agent applies the `ready-for-dev` label. | ||
|
|
||
| ## Quick Start (Operator) | ||
|
|
||
| For a first-time setup, complete these steps in order: | ||
|
|
||
| 1. Configure required secrets in **Settings → Secrets and variables → Actions**: | ||
| - `ANTHROPIC_API_KEY` and/or `GROQ_API_KEY` | ||
| - `AI_PR_TOKEN` (recommended for reliable PR/label/review writes) | ||
| 2. (Optional) Configure provider variables: | ||
| - `AI_PROVIDER` — `anthropic` or `groq`. Only needed when both keys are configured; Groq is the default. | ||
| - `ANTHROPIC_MODEL` — Anthropic model name (defaults to `claude-opus-4-7` if unset). | ||
| - `GROQ_MODEL` — Groq model name override for all stages (if unset, stage defaults from `config/models.yaml` are used: `generation`/`autofix` = `llama-3.3-70b-versatile`, `validation`/`review` = `qwen/qwen3-32b`). | ||
| - `GROQ_API_URL` — Groq endpoint URL (defaults to `https://api.groq.com/openai/v1/chat/completions` if unset). | ||
|
|
||
| ### Per-workflow environment variable matrix | ||
|
|
||
| All four workflows pass both provider key sets, so provider selection is driven entirely by which secrets are configured in the repository — no workflow-level override is needed. | ||
|
|
||
| | Workflow | Required secret(s) | Optional variables | Fallback | | ||
| |---|---|---|---| | ||
| | `validate-issue.yml` | `ANTHROPIC_API_KEY` or `GROQ_API_KEY` | `AI_PROVIDER`, `ANTHROPIC_MODEL`, `GROQ_MODEL`, `GROQ_API_URL` | Fails with clear error if neither key is present | | ||
| | `code-generation.yml` | `ANTHROPIC_API_KEY` or `GROQ_API_KEY` | `AI_PROVIDER`, `ANTHROPIC_MODEL`, `GROQ_MODEL`, `GROQ_API_URL` | Fails with clear error if neither key is present | | ||
| | `pr-review.yml` | `ANTHROPIC_API_KEY` or `GROQ_API_KEY` | `AI_PROVIDER`, `ANTHROPIC_MODEL`, `GROQ_MODEL`, `GROQ_API_URL` | Fails with clear error if neither key is present | | ||
| | `auto-fix-pr.yml` | `ANTHROPIC_API_KEY` or `GROQ_API_KEY` | `AI_PROVIDER`, `ANTHROPIC_MODEL`, `GROQ_MODEL`, `GROQ_API_URL` | Fails with clear error if neither key is present | | ||
|
|
||
| `AI_PR_TOKEN` is used only by `code-generation.yml`, `pr-review.yml`, and `auto-fix-pr.yml` for GitHub API write operations. | ||
|
|
||
| ## GitHub Actions PR Permission Requirement | ||
|
|
||
| If the run fails with: | ||
|
|
||
| `GitHub Actions is not permitted to create or approve pull requests.` | ||
|
|
||
| you have two supported options: | ||
|
|
||
| 1. Enable repository setting **Settings → Actions → General → Workflow permissions → Allow GitHub Actions to create and approve pull requests**. | ||
| 2. Set `AI_PR_TOKEN` and keep the setting disabled (recommended for stricter org policies). | ||
|
|
||
| ## Required Label | ||
|
|
||
| The validation workflow creates and manages these labels automatically: | ||
|
|
||
| - `ready-for-dev` — applied when issue quality is sufficient; triggers PR generation. | ||
| - `needs-refinement` — applied when the issue requires clearer acceptance criteria. | ||
|
|
||
| The PR review workflow creates and manages these labels automatically: | ||
|
|
||
| - `review-approved` — applied when the automated code review verdict is APPROVED. | ||
| - `changes-requested` — applied when the automated code review verdict is REQUEST_CHANGES. | ||
|
|
||
| All label names, colors, and descriptions are configurable in `config/labels.yaml`. | ||
|
|
||
| ## End-to-End Test | ||
|
|
||
| 1. Ensure secrets above are configured. | ||
| 2. Create a new GitHub issue using the feature or bug template, with a clear title and body. | ||
| 3. Open **Actions** and confirm run `Issue Validation Agent` starts. | ||
| 4. Once validation passes, confirm `Code Generation` starts automatically from the `ready-for-dev` label event. | ||
| 5. Verify logs for: | ||
| - prompt construction | ||
| - LLM API call success | ||
| - branch creation (`ai/issue-<number>`) | ||
| - PR creation | ||
| 6. Confirm PR details: | ||
| - title references the issue number/title | ||
| - body includes generated summary and `Closes #<number>` | ||
| - changed files are limited to the generated AI target paths (maximum 6 files) | ||
|
|
||
| ## Loop: `issue -> issue review` | ||
|
|
||
| The project now runs as a continuous loop rather than a one-shot generation: | ||
|
|
||
| 1. **Issue validation** (`validate-issue.yml`) reviews issue quality and applies `ready-for-dev` or `needs-refinement`. | ||
| 2. **Code generation** (`code-generation.yml`) starts only when `ready-for-dev` is applied and opens/updates a PR for that issue. | ||
| 3. **PR review** (`pr-review.yml`) runs on branch pushes, posts structured feedback, submits review status, and applies `review-approved` or `changes-requested`. | ||
| 4. **Auto-fix** (`auto-fix-pr.yml`) runs when `changes-requested` is applied, generates a targeted fix commit, and pushes it. | ||
| 5. The push from auto-fix re-triggers **PR review**, creating the iterative review loop. | ||
| 6. The loop ends when either: | ||
| - PR review returns `review-approved`, or | ||
| - auto-fix reaches 3 attempts and requests manual intervention. | ||
|
|
||
| ## End-to-End Control Flow | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| actor User | ||
| participant Issue | ||
| participant validate-issue.yml | ||
| participant code-generation.yml | ||
| participant pr-review.yml | ||
| participant auto-fix-pr.yml | ||
| participant PR | ||
|
|
||
|
|
||
| ``` | ||
|
|
||
| ## Structured Logging API (`scripts/lib/logger.mjs`) | ||
|
|
||
| All automation scripts share a structured JSON logger. Each line written to stdout/stderr is a valid JSON object. | ||
|
|
||
| ### Core functions | ||
|
|
||
| ```js | ||
| import { log, error, setLogContext, logStart, logEnd, logSummary } from './lib/logger.mjs'; | ||
| ``` | ||
|
|
||
| | Function | Output stream | `level` field | Description | | ||
| |---|---|---|---| | ||
| | `log(msg, data?)` | stdout | `info` | General informational event | | ||
| | `error(msg, data?)` | stderr | `error` | Error or warning event | | ||
| | `logSummary({ success, stepsCompleted, errors })` | stdout | `info` | Emits a `run_summary` entry at script exit | | ||
| | `logStart(step)` | — | — | Records start timestamp for a named step | | ||
| | `logEnd(step, result)` | stdout | `info` | Emits `step_end` with elapsed `durationMs` | | ||
|
|
||
| ### Log context | ||
|
|
||
| Call `setLogContext(fields)` once at startup to attach fields (e.g. `run_id`, `step`, `attempt`) to every subsequent `log` and `error` call. Per-call `data` fields override context fields with the same key. | ||
|
|
||
| ```js | ||
| setLogContext({ run_id: process.env.GITHUB_RUN_ID, step: 'auto-fix', attempt: 1 }); | ||
| log('Starting', { prNumber: 42 }); | ||
| // → {"level":"info","msg":"Starting","run_id":"…","step":"auto-fix","attempt":1,"prNumber":42} | ||
| ``` | ||
|
|
||
| ### Run summary | ||
|
|
||
| Emit a terminal summary in the `unhandledRejection` handler and at normal exit so log consumers can detect silent failures: | ||
|
|
||
| ```js | ||
| // on failure | ||
| logSummary({ success: false, stepsCompleted: ['labels'], errors: [err.message] }); | ||
|
|
||
| // on success | ||
| logSummary({ success: true, stepsCompleted: ['labels', 'diff', 'llm', 'write', 'label'], errors: [] }); | ||
| ``` | ||
|
|
||
| ### Step timing | ||
|
|
||
| ```js | ||
| logStart('llm-call'); | ||
| const raw = await callLLM(…); | ||
| logEnd('llm-call', 'ok'); | ||
| // → {"level":"info","msg":"step_end","step":"llm-call","result":"ok","durationMs":1234.5} | ||
| ``` | ||
|
|
||
| ## Minimum Test Coverage Policy | ||
|
|
||
| The following module must maintain **≥ 80% test coverage**, enforced in CI by `test.yml` via `c8 --check-coverage`: | ||
|
|
||
| - **Checkpoint resume** (`scripts/lib/checkpoint.mjs`): ≥ 80% across statements, branches, functions, and lines. Every distinct failure branch (ENOENT vs non-ENOENT in `readCheckpoint`, `mkdir` propagation in `writeCheckpoint`) must have a dedicated test case. | ||
|
|
||
| The following modules also maintain **≥ 80% test coverage**, each enforced by a dedicated `c8 --check-coverage` step in `test.yml`: | ||
|
|
||
| - **Config** (`scripts/lib/config.mjs`) | ||
| - **LLM client** (`scripts/lib/llm_client.mjs`) | ||
| - **Output writer** (`scripts/lib/output_writer.mjs`) | ||
|
|
||
| ## Auto-Fix Token Budget | ||
|
|
||
| The auto-fix stage constructs an LLM prompt from three sources — PR diff, review feedback, and current file contents. The total request size is capped to respect provider per-request limits (notably Groq on_demand's 12,000 TPM hard limit). | ||
|
|
||
| Three keys in `config/models.yaml` control the budget for the `autofix` stage: | ||
|
|
||
| | Key | Default | Description | | ||
| |---|---|---| | ||
| | `autofix_max_input_tokens` | `7400` | Hard ceiling on the total user-prompt tokens (wrapper + diff + feedback + files). Set to stay within `12000 − system_tokens − max_output_tokens`. The static wrapper text of `auto-fix-user.md` (~218 tokens) is deducted first; the remainder is divided among the three sections. Remove the key to use the full model context window (e.g. after upgrading to Groq Dev Tier or switching to Anthropic). | | ||
| | `autofix_diff_ratio` | `0.45` | Fraction of the section budget (after wrapper deduction) allocated to the PR diff. | | ||
| | `autofix_feedback_ratio` | `0.25` | Fraction of the section budget allocated to review feedback. The remainder goes to file contents. | | ||
|
|
||
| **Tuning for your provider tier:** | ||
|
|
||
| | Provider / Tier | Recommended `autofix_max_input_tokens` | | ||
| |---|---| | ||
| | Groq on_demand (`llama-3.3-70b-versatile`) | `7400` (default) | | ||
| | Groq Dev Tier | Remove the key (no cap needed — 12k TPM limit applies per-minute, not per-request) | | ||
| | Anthropic (`claude-opus-4-7`) | Remove the key (200k context window; no per-request TPM limit) | | ||
|
|
||
| The `token_estimate` log line emitted by `auto_fix_pr.mjs` shows the actual token counts for each section: | ||
|
|
||
| ```json | ||
| {"level":"info","msg":"token_estimate","system":459,"wrapper":218,"diff":3231,"feedback":1795,"files":2156,"max_tokens":4096,"total":11955} | ||
| ``` | ||
|
|
||
| Monitor this to detect systematic truncation of diff or file contents. | ||
|
|
||
| ## Checkpoint Resume | ||
|
|
||
| Critical job outputs are persisted to `./checkpoints/<runId>/<step>.json` via `scripts/lib/checkpoint.mjs` and exchanged between jobs as GitHub Actions artifacts. This allows a re-triggered run to resume from the last successful step rather than starting from scratch. | ||
|
|
||
| ### RunId naming conventions | ||
|
|
||
| | Workflow chain | RunId format | Artifact name | | ||
| |---|---|---| | ||
| | `validate-issue` → `code-generation` | `issue-<number>` | `checkpoints-issue-<number>` | | ||
| | `pr-review` → `auto-fix-pr` | `pr-<number>` | `checkpoints-pr-<number>` | | ||
|
|
||
| The `CHECKPOINT_RUN_ID` environment variable overrides the default; each script falls back to deriving the ID from `ISSUE_NUMBER` or the resolved PR number if the variable is absent. | ||
|
|
||
| ### Step names written per job | ||
|
|
||
| | Job script | Step name | Persisted fields | | ||
| |---|---|---| | ||
| | `validate_issue.mjs` | `validate` | `valid`, `score` | | ||
| | `generate_issue_change.mjs` | `generate` | `summary`, `outputPaths` | | ||
| | `pr_review.mjs` | `review` | `isApproved`, `prNumber` | | ||
| | `auto_fix_pr.mjs` | `autofix` | `prNumber`, `attempt`, `outputPaths` | | ||
|
|
||
| ### Prerequisite enforcement | ||
|
|
||
| `code-generation` (when triggered by the automated pipeline via `validate_run_id` dispatch input) and `auto-fix-pr` both fail explicitly with `::error::` if the expected upstream checkpoint file is not present, preventing execution in an incoherent state. | ||
|
|
||
| ### Cross-workflow artifact download | ||
|
|
||
| - `validate-issue` passes its `GITHUB_RUN_ID` as the `validate_run_id` workflow-dispatch input when triggering `code-generation`. The generate job uses this run-id to download the artifact. | ||
| - `auto-fix-pr` uses `gh api repos/{owner}/{repo}/actions/artifacts?name=checkpoints-pr-{PR_NUMBER}` to locate the latest non-expired checkpoint artifact and downloads it with `curl`. | ||
|
|
||
| ### Artifact retention | ||
|
|
||
| GitHub Actions artifact retention applies (default 90 days). Old checkpoint artifacts for the same issue or PR number are overwritten (`overwrite: true`) on each new run. | ||
|
|
||
| ### Test coverage requirement | ||
|
|
||
| `scripts/lib/checkpoint.mjs` and all code paths that call `writeCheckpoint`/`readCheckpoint` must maintain **≥ 80% test coverage**. Every distinct failure branch (e.g. ENOENT vs non-ENOENT in `readCheckpoint`, `mkdir` propagation in `writeCheckpoint`) must have a dedicated test case. | ||
|
|
||
| ## Startup Fail-Fast Validation (automation entrypoints) | ||
|
|
||
| Automation scripts must fail before network calls when required startup inputs are invalid: | ||
|
|
||
| - **Environment**: required vars are validated synchronously at process start (`GITHUB_TOKEN`, `GITHUB_REPOSITORY`, `GITHUB_EVENT_PATH`, provider API key, etc.). | ||
| - **Prompts**: prompt files are loaded and validated as existing + non-empty at startup, with explicit file-path errors when missing/empty. | ||
| - **GitHub payload**: required fields are validated with path-based errors: | ||
| - PR number: `pull_request.number` or fallback `issue.number`. | ||
| - Branch reference (when needed): `pull_request.head.ref` or fallback `ref`. | ||
| - **Provider payload parsing**: response-shape failures include concrete expected paths: | ||
| - Anthropic: `content[0].text` | ||
| - Groq: `choices[0].message.content` | ||
|
|
||
| ## CI Coverage Enforcement | ||
|
|
||
| The repository enforces a minimum test coverage policy through CI using `c8 --check-coverage`. The following modules must maintain **≥ 80% test coverage**, each enforced by a dedicated step in `test.yml`: | ||
|
|
||
| - **Checkpoint resume** (`scripts/lib/checkpoint.mjs`) | ||
| - **Configuration** (`scripts/lib/config.mjs`) | ||
| - **LLM client** (`scripts/lib/llm_client.mjs`) | ||
| - **Output writer** (`scripts/lib/output_writer.mjs`) | ||
| # Code Generation | ||
| ## Introduction | ||
| This document outlines the code generation process. | ||
| ## Automation Documentation | ||
| The PostToolUse hook is used to trigger tests on script/workflow edits. This hook is configured in .claude/settings.json and is triggered when a file in the scripts/ or .github/workflows/ directories is edited. | ||
| ## Test Coverage Requirements | ||
| The test coverage requirements for automation workflows are as follows: at least 80% of the code must be covered by unit tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| import { test } from '@jest/globals'; | ||
| import fs from 'fs'; | ||
| import path from 'path'; | ||
| import { execSync } from 'child_process'; | ||
|
|
||
| const testFilePath = path.join(__dirname, 'test-file.txt'); | ||
| const scriptPath = path.join(__dirname, '../..'); | ||
|
|
||
| beforeEach(() => { | ||
| fs.writeFileSync(testFilePath, 'test content'); | ||
| }); | ||
|
|
||
| afterEach(() => { | ||
| fs.unlinkSync(testFilePath); | ||
| }); | ||
|
|
||
| test('PostToolUse hook triggers tests for scripts/ directory', () => { | ||
| const result = execSync(`node --test ${scriptPath}/scripts/tests/*.test.mjs`, { | ||
| env: { | ||
| TOOL_INPUT_FILE_PATH: path.join(scriptPath, 'scripts/test-file.txt'), | ||
| }, | ||
| }); | ||
| expect(result.toString()).toContain('Running test suite...'); | ||
| }); | ||
|
|
||
| test('PostToolUse hook triggers tests for .github/workflows/ directory', () => { | ||
| const result = execSync(`node --test ${scriptPath}/scripts/tests/*.test.mjs`, { | ||
| env: { | ||
| TOOL_INPUT_FILE_PATH: path.join(scriptPath, '.github/workflows/test-file.txt'), | ||
| }, | ||
| }); | ||
| expect(result.toString()).toContain('Running test suite...'); | ||
| }); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this repo,
node --test scripts/tests/*.test.mjscurrently takes essentially the full limit (duration_ms 59553.974856, about 59.99s, when run from/workspace/autonomous-dev-loop). Claude Code's hook docs definetimeoutas seconds before canceling the hook, so this 60s cap leaves no margin for normal machine variance or hook startup overhead; on edits underscripts/or.github/workflows/the test hook can be canceled before reporting the regression it is meant to catch. Increase the timeout or avoid running the full suite synchronously here.Useful? React with 👍 / 👎.