feat(miner): add a --full-execution dry-run mode to the cross-repo harness

[bitfathers94]

Summary

Extends the cross-repo evaluation harness (#4788) from readiness-only to a full --full-execution mode
(#7634): it drives the live discover → plan → code → test loop against a curated subset of the existing
benchmark repos, generates a real diff, and runs each target repo's own build + test commands locally —
then classifies the outcome. This is the missing signal behind #4810's launch-readiness bar: not just "can the
miner plan for this repo" but "does the miner actually produce working, correct code for it".

Dry-run only, same safety posture as the readiness harness, one step further. The coding agent edits a
throwaway local clone; the harness captures the diff with git and runs the repo's own tests. There is no live
GitHub PR submission, no write access to the third-party repos, and no credentials beyond the local clone.

What changed

• lib/cross-repo-evaluation.ts — new executeRepoAttempt / runCrossRepoExecution pipeline that gates on
  the existing readiness check first, then runs the live attempt and classifies it. The coding-agent step, build,
  and test runners are all injectable (real spawnSync defaults for build/test).
• Extended failure taxonomy (added to the existing CROSS_REPO_FAILURE_CATEGORY, not replacing it):
  execution_no_diff, execution_compile_gap, execution_test_failure, execution_noop_diff. Results flow
  through the same formatCrossRepoEvaluationReport / summarizeCrossRepoEvaluation output.
• scripts/cross-repo-evaluation.mjs — --full-execution flag wiring a real, driver-backed executor
  (constructProductionCodingAgentDriver + git diff capture) plus runCrossRepoExecutionCli.
• benchmarks/cross-repo/manifest.json — a fullExecution opt-in flag marks the subset (2 Node repos:
  expressjs/express, lodash/lodash); a --repo filter overrides it.
• docs/cross-repo-evaluation.md — documents the new mode, taxonomy, dry-run safety, and the manifest flag.

Testing

packages/loopover-miner/** is outside coverage.include, but the new logic has real unit tests following this
package's existing conventions (test/unit/miner-cross-repo-evaluation.test.ts): every taxonomy branch
(readiness pass-through, no-diff, compile gap, test failure, no-op, end-to-end pass), the default spawn-based
build/test runners via an injected command runner, subset/--repo filtering, manifest parsing of the new flag,
and the CLI's real driver-backed executor (with an injected fake driver/spawn).

vitest run test/unit/miner-cross-repo-evaluation.test.ts   # 56 passed
npm run typecheck && npm run build --workspace @loopover/miner   # clean

Closes #7634

[superagent-security]

Superagent found 1 security concern(s).

[superagent-security]

P1: Arbitrary shell command execution from cloned repo stack detection

defaultRunLocalCommand passes untrusted repo commands directly to sh -c, enabling arbitrary code execution from cloned repository metadata.

Avoid sh -c wrapper; run commands directly as argument arrays, or validate/sanitize before shell execution.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name="packages/loopover-miner/lib/cross-repo-evaluation.ts">
<violation number="1" location="packages/loopover-miner/lib/cross-repo-evaluation.ts:492">
<priority>P1</priority>
<title>Arbitrary shell command execution from cloned repo stack detection</title>
<evidence>defaultRunLocalCommand uses spawnSync("sh", ["-c", command], ...) where command comes directly from stack.buildCommand or stack.testCommand, inferred from cloned repository metadata (e.g., package.json scripts). A malicious benchmark repo can inject shell metacharacters to execute arbitrary commands on the host evaluating the harness.</evidence>
<recommendation>Validate and sanitize commands before passing to sh -c. Prefer running commands directly via their binary (e.g., "npm test" as ["npm", "test"]) rather than through a shell wrapper. Consider sandboxing or restricting the execution environment for untrusted repos.</recommendation>
</violation>
</file>

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

❌ Patch coverage is 79.71014% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.49%. Comparing base (22c2c4a) to head (62d411e).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...ckages/loopover-miner/lib/cross-repo-evaluation.ts	79.71%	0 Missing and 14 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7641      +/-   ##
==========================================
- Coverage   88.51%   88.49%   -0.02%     
==========================================
  Files         724      724              
  Lines       76001    76069      +68     
  Branches    22619    22652      +33     
==========================================
+ Hits        67272    67319      +47     
- Misses       7681     7688       +7     
- Partials     1048     1062      +14     

Flag	Coverage Δ
shard-1	35.77% <79.71%> (+0.05%)	⬆️
shard-2	32.93% <0.00%> (-0.06%)	⬇️
shard-3	28.55% <0.00%> (-0.08%)	⬇️
shard-4	40.46% <0.00%> (-0.16%)	⬇️
shard-5	33.96% <0.00%> (-0.05%)	⬇️
shard-6	35.92% <0.00%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ckages/loopover-miner/lib/cross-repo-evaluation.ts	94.11% <79.71%> (-5.89%)	⬇️

... and 1 file with indirect coverage changes

[loopover-orb]

Caution

🛑 LoopOver review result - reject/close recommended

_{Review updated: 2026-07-21 06:32:11 UTC}

8 files · 1 AI reviewer · 1 blocker · CI failing · blocked

🛑 Suggested Action - Reject/Close

Review summary
This adds a new full-execution mode to the cross-repo evaluation harness (readiness gate → live coding attempt → compile → test → no-op check), threading through .ts, its compiled .js/.d.ts, the CLI script, manifest opt-in flags, and docs. The core `executeRepoAttempt` pipeline is logically sound and well-staged, the default command runner correctly avoids shell injection by using `shell:false` with argv splitting, and the whole thing stays dry-run/no-write as claimed. The PR ships compiled .js/.d.ts artifacts alongside the .ts source (three parallel copies of the same logic), which is consistent with this repo's existing pattern but is a real maintenance/drift risk, and coverage is short of the ~97% patch bar (79.71% reported).

Blockers

• codecov/patch is at 79.71% (target 99%) and validate/validate-code are failing — the PR does not meet this repo's coverage and validation gates as submitted, so changed branches in the new pipeline (e.g. some of the compile/test/no-diff branches in `executeRepoAttempt`, packages/loopover-miner/lib/cross-repo-evaluation.ts:558-660) are not demonstrably covered.
• The description states this closes/relates to Extend the cross-repo evaluation harness from readiness-only to full execution on benchmark repos #7634 and cites feat(miner): add a --full-execution dry-run mode to the cross-repo harness #7641 for the shell-injection fix, but the issue-linkage requirement for external contributor PRs isn't verifiable from the diff alone — confirm an eligible open issue is linked before merge.

Nits — 5 non-blocking

• packages/loopover-miner/lib/cross-repo-evaluation.ts:495 `executeRepoAttempt` is a long, deeply-nested (depth 5) function with ~18 branches per the complexity check — consider extracting the compile/test/no-op stages into named helper functions the way `runStackCommand` already factors out command execution.
• The magic issue numbers (`7634`, `7641`) scattered through comments in .ts/.js/.d.ts are fine as historical references but add noise; a single top-of-file comment block naming the taxonomy rationale would reduce duplication across the three parallel files.
• packages/loopover-miner/lib/cross-repo-evaluation.ts:520 the `maxTurns` default of `30` is a bare literal — consider a named `DEFAULT_MAX_TURNS` constant since it's referenced in both the CLI and this module per the docs.
• Shipping hand-synced .js/.d.ts artifacts (not build-generated in this diff) is a drift risk — a future edit to the .ts that isn't mirrored to .js will silently diverge; worth confirming a build step regenerates these rather than manual dual-editing.
• Add/extend unit tests around the `execution_compile_gap` and `execution_test_failure` branches in packages/loopover-miner/lib/cross-repo-evaluation.ts:558-596 to close the coverage gap flagged by codecov/patch.

Why this is blocked

• codecov/patch is at 79.71% (target 99%) and validate/validate-code are failing — the PR does not meet this repo's coverage and validation gates as submitted, so changed branches in the new pipeline (e.g. some of the compile/test/no-diff branches in `executeRepoAttempt`, packages/loopover-miner/lib/cross-repo-evaluation.ts:558-660) are not demonstrably covered.

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. codecov/patch is at 79.71% \(target 99%\) and validate/validate-code are failing — the PR does not meet this repo's coverage and validation gates as submitted, so changed branches in the new pipeline \(e.g. some of the compile/test/no-diff branches in \`executeRepoAttempt\`, packages/loopover-miner/lib/cross-repo-evaluation.ts:558-660\) are not demonstrably covered.

CI checks failing

• validate
• codecov/patch — 79.71% of diff hit (target 99.00%)
• validate-code

Decision drivers

• ❌ Code review — 1 blocker (1 reviewer)
• ❌ Gate result — Blocking (Repo-configured hard blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#7634
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	❌ 8/20	High review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 71 registered-repo PR(s), 45 merged, 0 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	bitfathers94; Gittensor profile; 71 PR(s), 0 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: moderate

Linked issue satisfaction

Addressed
The PR adds `executeRepoAttempt`/`runCrossRepoExecution` implementing the live discover→plan→code→build→test pipeline gated on existing readiness, wires a `--full-execution` CLI flag with a manifest opt-in flag covering 2 of the 6 repos (express, lodash), extends the existing failure taxonomy with the four requested execution-specific categories flowing through the same report format, and updates

Review context

• Author: bitfathers94
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 71 PR(s), 0 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Add a concise scope and risk note.
• Then work through the remaining 1 step in the Signals table above.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[loopover-orb]

LoopOver is closing this pull request on the maintainer's behalf (CI is failing (validate, codecov/patch, validate-code); AI reviewers agree on a likely critical defect: codecov/patch is at 79.71% (target 99%) and validate/validate-code are failing — the PR does not meet this repo's coverage and validation gates as submitted, so changed branches in the new pipeline (e.g. some of the compile/test/no-diff branches in `executeRepoAttempt`, packages/loopover-miner/lib/cross-repo-evaluation.ts:558-660) are not demonstrably covered.). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.