Skip to content

feat(miner): add cross-repo --full-execution dry-run mode#7639

Closed
kai392 wants to merge 3 commits into
JSONbored:mainfrom
kai392:feat/cross-repo-full-execution-7634
Closed

feat(miner): add cross-repo --full-execution dry-run mode#7639
kai392 wants to merge 3 commits into
JSONbored:mainfrom
kai392:feat/cross-repo-full-execution-7634

Conversation

@kai392

@kai392 kai392 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds --full-execution to the existing cross-repo evaluation CLI/lib so at least two tagged benchmark repos can run local discover→plan→code→build→test without opening forge PRs (Closes #7634).
  • Extends the failure taxonomy with plan_formed_compile_failed, compiled_tests_failed, tests_passed_noop_diff, and execution_abandon, keeping the same report format.
  • Documents the mode and covers the new branches in test/unit/miner-cross-repo-evaluation.test.ts.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint — no workflow changes
  • npm run typecheck — miner package rebuilt via tsc -p packages/loopover-miner/tsconfig.json
  • Focused unit tests: vitest run against test/unit/miner-cross-repo-evaluation.test.ts (49 passed). Package path is outside Codecov src/** patch gate for apps; miner lib coverage still exercised by the new tests.
  • npm run test:workers — unrelated
  • npm run build:mcp — unrelated
  • npm run test:mcp-pack — unrelated
  • npm run ui:openapi:check — unrelated
  • npm run ui:lint — unrelated
  • npm run ui:typecheck — unrelated
  • npm run ui:build — unrelated
  • npm audit --audit-level=moderate — skipped (no dependency changes)
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • No workflow/UI/MCP/dependency changes; validation focused on miner cross-repo harness unit tests + local tsc emit for the dual-published lib.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

N/A — miner CLI/lib/docs only; no visible UI changes.

Notes

  • Dry-run only: default coding seam abandons unless runCodingAttempt is injected or LOOPOVER_MINER_FULL_EXECUTION_STUB=1 is set. Never opens PRs against benchmark repos.
  • Manifest tags expressjs/express and lodash/lodash with fullExecution: true (≥2 of 6).

Closes #7634

@kai392
kai392 requested a review from JSONbored as a code owner July 21, 2026 05:39

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

export function defaultRunShellCommand(input: { command: string; cwd: string }): CrossRepoShellCommandResult {
const result = spawnSync(input.command, {
cwd: input.cwd,
shell: true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: spawnSync with shell: true executes benchmark repo commands without validation or timeout

spawnSync runs external benchmark repo build/test commands with shell: true, inheriting the full parent environment and without a timeout.

Add a timeout to spawnSync, validate or sanitize commands before execution, and pass a minimal environment to child processes.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name="packages/loopover-miner/lib/cross-repo-evaluation.ts">
<violation number="1" location="packages/loopover-miner/lib/cross-repo-evaluation.ts:460">
<priority>P2</priority>
<title>`spawnSync` with `shell: true` executes benchmark repo commands without validation or timeout</title>
<evidence>`defaultRunShellCommand` uses `spawnSync(input.command, { cwd: input.cwd, shell: true, encoding: "utf8", env: process.env })` where `input.command` comes from `stack.buildCommand` or `stack.testCommand` detected from cloned benchmark repositories. This passes the full parent environment to child processes and lacks a timeout, allowing a compromised benchmark repo to execute arbitrary shell commands or hang the evaluation process indefinitely.</evidence>
<recommendation>Add a `timeout` to `spawnSync` (e.g., `timeout: 300000` for 5 minutes), validate or sanitize `command` before execution, and consider using `shell: false` with pre-parsed arguments when possible. Pass a sanitized or minimal environment instead of `process.env` to reduce secret exposure from CI environments.</recommendation>
</violation>
</file>

@superagent-security superagent-security Bot added the pr:flagged PR flagged for review by security analysis. label Jul 21, 2026
Extend the existing harness so benchmark repos can run local code/build/test without forge PR writes (Closes JSONbored#7634).

Co-authored-by: Cursor <cursoragent@cursor.com>
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.07143% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.48%. Comparing base (4d9b9d1) to head (49d79c0).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...ckages/loopover-miner/lib/cross-repo-evaluation.ts 66.07% 9 Missing and 10 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7639      +/-   ##
==========================================
- Coverage   88.51%   88.48%   -0.03%     
==========================================
  Files         724      724              
  Lines       75992    76045      +53     
  Branches    22618    22643      +25     
==========================================
+ Hits        67263    67290      +27     
- Misses       7681     7697      +16     
- Partials     1048     1058      +10     
Flag Coverage Δ
shard-1 35.75% <66.07%> (+0.03%) ⬆️
shard-2 32.93% <0.00%> (-0.05%) ⬇️
shard-3 28.56% <0.00%> (-0.07%) ⬇️
shard-4 40.48% <0.00%> (-0.15%) ⬇️
shard-5 33.96% <0.00%> (-0.04%) ⬇️
shard-6 35.93% <0.00%> (-0.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ckages/loopover-miner/lib/cross-repo-evaluation.ts 91.47% <66.07%> (-8.53%) ⬇️

... and 1 file with indirect coverage changes

Co-authored-by: Cursor <cursoragent@cursor.com>
@JSONbored

Copy link
Copy Markdown
Owner

Superagent findings, fix + resubmit.

@JSONbored JSONbored closed this Jul 21, 2026
…T.md

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr:flagged PR flagged for review by security analysis.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend the cross-repo evaluation harness from readiness-only to full execution on benchmark repos

3 participants