feat(algorithms)!: unify PyTorch runtime contracts

[jiangxt2]

Description

This PR replaces Tributo's divergent PyTorch algorithm integration paths with one versioned contract and one Core-owned Ray Train runtime. TorchRecipe covers Core-owned training loops, RayTorchAdapter covers framework-owned worker loops, and both execute through DistributionStrategy.RAY_TRAIN_TORCH and runtime ID tributo.ray_train_torch under torch_runtime_api_version=1.

The previous TorchTrainingRecipe and TrainingRecipeV2 surfaces both reached the same training implementation, while complex PyTorch algorithms bypassed that path by constructing their own TorchTrainer. The new boundary makes TorchPolicy.execution_plan the single source of truth and assigns Trainer construction, role routing, collective alignment, checkpoint transport, execution evidence, and Bundle publication to Core; algorithm Wheels retain model, loss, metric, framework-adaptation, and artifact-signature responsibilities.

Key changes:

• Add versioned TorchPolicy, single-stage and component-stage execution plans, role-routed datasets, typed runtime and stage contexts, and public TorchRecipe and RayTorchAdapter SPIs.
• Add the unified tributo.ray_train_torch runtime with Torch-only preflight, invocation-local one-shot lease transfer, Core-owned TorchTrainer orchestration, exact split and bounded replication routing, and explicit final-stage selection.
• Add normalizer-aware loss accumulation, composite-loss reducer invocation, explicit metric reducers, rank-aligned zero contributions for optional evaluation roles, and typed execution evidence bound to the resolved Policy and plan.
• Add stage-scoped run identities, checkpoint descriptors, credential-free worker control and recovery envelopes, payload digests, retry cursor and scheduler state, commit markers, and staged local or remote persistence.
• Replace ray-torch-recipe-v1 with ray-torch-v1, split Recipe reconstruction from Adapter export-source handling, and route both through the existing BundleExportService.
• Migrate Core fixtures, public API inventory, architecture documentation, official algorithm identities, and regression tests to the new PyTorch contract.

This is intentionally one atomic implementation change because the public Policy, SPI, Runtime, Checkpoint, Evidence, and Source Provider identities must move together; merging an intermediate subset would leave installed PyTorch descriptors without a valid execution or export path.

Related issues

• Follows up on feat(training): add Ray-first PyTorch recipe execution #73, which introduced the previous Ray-first PyTorch Recipe path.
• Builds on the official algorithm Wheel boundary from feat: complete distributed algorithm wheel ecosystem #81 and the 37-entry-point certification baseline from test: certify official distributed algorithms #85.

Additional information

Cross-repository coordination

• Core implementation: 878c49f.
• Companion algorithm migration: 3d7d838, already present on tributo-algorithms/master.
• Candidate Core Wheel provenance: Pending. No candidate Wheel has been built from 878c49f; existing candidate Wheels predate the final Core commit and are not valid evidence for this PR.
• The source merge order is therefore algorithm migration first and this Core PR second. The Core PR must not merge until joint validation succeeds; package publication order is handled separately from this source merge.

Breaking changes and migration

• Remove TorchTrainingRecipe, TrainingRecipeV2, AlgorithmBuilder.from_torch_recipe(), and AlgorithmBuilder.from_training_recipe_v2().
• Remove the PyTorch-specific legacy Recipe strategy, execution mode, topology, runtime identity, and ray-torch-recipe-v1 source-provider identity.
• Replace those entry points with TorchRecipe, RayTorchAdapter, AlgorithmBuilder.from_torch(), AlgorithmBuilder.from_torch_adapter(), RAY_TRAIN_TORCH, tributo.ray_train_torch, and ray-torch-v1.
• Require PyTorch algorithm Wheels to declare TorchPolicy, the execution plan, role routing, metric reducers, state layout, and the independent Torch Runtime API version.
• Reject pre-migration PyTorch recovery checkpoints and envelopes that do not bind the new Policy, plan, stage, input, implementation, and payload identities; no compatibility reader or automatic upgrade is provided.
• Rename the official PyTorch entry points dnn.recipe_v2, pu.recipe_v2, lstm_classifier.recipe_v2, and gru_classifier.recipe_v2 to their algorithm identities and remove the .recipe_v2 suffix from the LSTM and GRU implementation IDs.

The companion tributo-algorithms migration must be released in coordination with this Core change. Existing PyTorch Wheels implementing the removed interfaces are not compatible with the new runtime.

Compatibility boundary

• The breaking boundary is limited to PyTorch algorithm contracts, identities, checkpoints, and the Torch Source Provider.
• MapReduce, Joblib estimator, parallel ensemble, iterative optimization, XGBoost collective, and non-PyTorch framework-native strategies retain their existing contracts and runtime IDs.
• Existing Bundle Manifest, ONNX Runtime, Ray Data batch inference, Ray Serve, Data Provider/Binding, WriteGateway, ResultSink, MLflow, OperationStore, Outbox, Hook, Broker, and Streaming contracts are not migrated.
• The inference/kernel.py change is a backward-compatible materialization fix for higher-rank nested object-array columns; it does not migrate the public inference request, binding, manifest, or runtime contracts.
• Previously published ONNX Bundles remain readable; recovery checkpoints are not Bundles and follow the explicit incompatibility described above.
• Core imports and descriptor-only discovery remain free of eager Torch, PyG, Transformers, and algorithm implementation imports.
• No third-party dependency is added or changed.

Local static and changed-test validation is complete; Docker Ray IT, companion Wheel conformance, and external cluster gates remain pending before merge.

Validation completed for commit 878c49f

• uv run --locked --no-sync python scripts/pr-precheck.py — passed all mandatory format, dependency, CI-policy, API-stability, suppression, commit, hygiene, changed-test, CI-parity, and documentation layers on the final source tree.
• uv run --locked --no-sync python scripts/pr-precheck.py --skip-tests — passed after the signed commit was created, confirming the final commit message, DCO trailer, static gates, and documentation gates without repeating the completed changed-test run.
• Pre-commit Ruff format and lint hooks — passed.
• git diff --check — passed.
• The pre-check reported 116 non-blocking Python Safety warnings; these were reviewed and are primarily function-local Torch or Ray imports that preserve lightweight module loading. The reported broad exception in inference/kernel.py predates this diff.

Pending validation before merge

• Pending: Run the repository's full non-external unit suite for Python 3.12 and 3.13 through the PR CI matrix.
• Pending: Run source-free 37-entry-point conformance in an external clean environment using the recorded Core and 15 algorithm Wheel SHA-256 values and verified non-editable installation paths.
• Pending: Run the final distributed-algorithm-cluster manual gate on exact Core revision 878c49f and algorithm revision 3d7d838 to cover multi-worker Ray Jobs training, retry and cross-node recovery, Tune checkpoint behavior, Bundle publication, and Ray Data inference without skipped tests.
• Pending: Review the additional manual-external suites selected by scripts/ci_test_plan.py for the pyproject.toml, runtime, export, inference, and documentation changes, and record each suite as passed or explicitly not required with a concrete scope reason.
• Pending: Do not issue GPU, NCCL, PyG, Transformers, FSDP, DeepSpeed, or TorchRec support claims without separately approved environment-specific gates.

Earlier development runs are useful diagnostic evidence but are not listed as final validation because the committed recovery and staging paths changed afterward.

Reviewer focus

• Verify that every shared Builder, Planner, Dispatcher, Receipt, Support, Bundle, and inference change is selected only by a Torch discriminant and does not alter the 24 non-PyTorch algorithm identities, contracts, routing, errors, or lifecycle.
• Verify that TorchPolicy.execution_plan is the only stage and routing declaration and that neither Recipe nor Adapter can create a nested Trainer or provide a competing execution plan.
• Verify the normalizer-weighted accumulation formula, composite reducer trust boundary, collective ordering, AMP unscale and gradient scaling order, zero-normalizer rejection, and metric reducer mapping.
• Verify preflight lease ownership, run and invocation identity binding, checkpoint source priority, credential isolation, payload digest validation, recovery cursor state, commit-marker enforcement, and idempotent staged persistence.
• Verify that completed and active component-stage recovery reconstructs evidence for the declared plan and that only final_stage_id can supply the exported Bundle.
• Verify the ray-torch-v1 Recipe and Adapter export branches, Artifact Plan binding, required-artifact failure behavior, and composition_digest propagation.
• Verify that the nested object-array inference adjustment preserves declared higher-rank tensor input shapes without changing scalar or ordinary vector bindings.

Non-goals

• This PR does not implement or claim FSDP, DeepSpeed, TorchRec, Lightning, Hugging Face Trainer, or native PyTorch serving support.
• This PR does not add a second DataLoader, distributed checkpoint store, cluster manager, deployment controller, or Bundle format.
• This PR does not remove global inference or serving compatibility paths or refactor unrelated Data, MLflow, ResultSink, OperationStore, Broker, or Streaming modules.