Split deterministic v1 tests from live E2Es

[xeophon]

Overview

Separate deterministic v1 coverage from live, provider-dependent end-to-end evaluations while preserving both test surfaces.

Changes

• Keep deterministic v1 tests on the full Python 3.11, 3.12, and 3.13 matrix through the existing e2e marker boundary.
• Run live v1 evaluations in a dedicated Python 3.12 job with bounded test concurrency and a job timeout.
• Keep Prime and Modal runtime placements local-only, matching the existing CI boundary.
• Scope provider credentials to the live job and skip that job for fork pull requests where secrets are unavailable.
• Remove the standalone Prime CLI installation from the deterministic matrix.
• Document the deterministic and live pytest selectors alongside the v1 fixtures.

Impact

Deterministic regressions remain covered across every supported Python version, while provider, runtime, server, and dynamic-tool behavior stays visible in a dedicated CI result with controlled resource usage.

Note

Split deterministic v1 tests from live E2E tests in CI

• Deterministic v1 tests (marked not e2e) now run in the main CI matrix job across all Python versions; the prime package install and related secrets are removed from this job.
• Live E2E tests (marked e2e and not prime and not modal) run in a new v1-e2e job gated on repository/actor conditions, targeting Python 3.12 only, with a 120-minute timeout and two parallel workers.
• conftest.py comments are updated to document the two invocation patterns.

^{Macroscope summarized 2951c19.}

---

Note

Low Risk
Changes are limited to GitHub Actions wiring and v1 test documentation; no application runtime or auth logic is modified.

Overview
CI now runs v1 tests in two lanes instead of one combined tests/v1 step on the Python matrix.

The main Verifiers job keeps the legacy suite and adds deterministic v1 coverage with -m "not e2e" on Python 3.11–3.13. It no longer installs Prime or injects PRIME/HF secrets, and the workflow sets permissions: contents: read.

A new v1-e2e job runs live evaluations once on Python 3.12 with -m "e2e and not prime and not modal", 2 workers, and a 120-minute timeout. Provider secrets live only in that job, and it is skipped for fork PRs (and Dependabot) where secrets are unavailable.

tests/v1/conftest.py docs now describe the not e2e matrix selector vs the live E2E command.

^{Reviewed by Cursor Bugbot for commit 2951c19. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2951c19. Configure here.}

[cursor]

HF token removed from matrix job

Medium Severity

HF_TOKEN was moved exclusively onto the live v1-e2e job, but the matrix test job still downloads Hugging Face tokenizers in tests/test_renderer_client.py and tests/test_renderer_e2e.py. Without the token, those Hub fetches run unauthenticated and can hit rate limits or gated-model failures on cache miss, while the live job that holds the token does not exercise those paths.

Additional Locations (1)

• .github/workflows/test.yml#L73-L77

 

^{Reviewed by Cursor Bugbot for commit 2951c19. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

CI workflow change that splits test jobs. An unresolved review comment identifies that HF_TOKEN removal from the matrix job may cause intermittent test failures due to rate limiting - this concern warrants human review to verify whether deterministic tests actually require authenticated HF access.

^{You can customize Macroscope's approvability policy. Learn more.}