split val metrics/viz - #501
Draft
aidang3019 wants to merge 23 commits into
Draft
Conversation
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
aidang3019
force-pushed
the
aidan/eval_gating
branch
from
July 8, 2026 00:45
f76e5c7 to
548ea50
Compare
This was referenced Jul 8, 2026
aidang3019
force-pushed
the
aidan/eval_gating
branch
from
July 10, 2026 03:37
548ea50 to
7dff7c6
Compare
aidang3019
force-pushed
the
aidan/eval_gating
branch
from
August 28, 2026 16:06
7dff7c6 to
db5533a
Compare
aidang3019
force-pushed
the
aidan/pi-6d
branch
from
August 28, 2026 16:06
282c2f6 to
9e1f401
Compare
… epochs - trainer/ddp_pi: check_val_every_n_epoch 10 -> 20 (80 val batches inherited from trainer/default; valid loader shuffle pinned false in mecka_all_pi_6d) - evaluator/eval_pi: viz_every_n_epochs 200 -> 100 - _should_viz now gates on (current_epoch + 1) % viz_every_n_epochs, matching Lightning's (current_epoch + 1) % check_val_every_n_epoch val condition — the plain current_epoch % n never coincided with a validation epoch unless check_val_every_n_epoch == 1, so viz never fired Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mpts One wholly-out-of-bounds episode (692ea23dc621d7f4aac3aaa1) exhausted the within-episode retry pool, killed rank 3, and NCCL-timeouted the other 7 ranks 8h into the mecka-all run. Retries now leave the failing episode after 25 attempts; 1000 consecutive failures still raise loudly. Also 6->8 dataloader workers; dedupe two accidentally duplicated tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bake the run cadence into the configs (ddp_pi check_val 20->50;
checkpoints every_n_epochs 50->100, save_top_k -1->3, monitor step/max)
and drop the now-redundant sbatch CLI overrides ('+' adds would error).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y 50 epochs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… ckpt handoff Matrix probe 10923198 exonerated the config: 8-rank backward is 2.1s on healthy nodes; both 12.6/11.4 s/step runs sat on atl1-1-01-009-16-0. HANDOFF_JOB/RESUME_DIR cancel the running lower-GPU job at start time and resume from its freshest checkpoint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nions Euler wrap at ±pi inflates cam-frame MSE: +pi-eps vs -pi+eps scores ~(2pi)^2 per dim. Wrap rotation-dim errors before squaring (12/14-wide YPR layouts only; 6D widths untouched). Unwrapped values stay logged as *_nowrap to quantify the inflation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ride) Distributed val deals unshuffled indices round-robin, so rank 0 renders every world_size-th source frame; writing at the raw 30fps produced a world_size-times timelapse (8x on the current run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Val metrics (eval_pi): xyz/ypr MSE splits (native + cam), Frechet Gaussian over time (avg/min/max), reverse-KL + best/mean/worst-of-M + sample diversity from M stochastic rollouts (sample_action_chunks, gated on config.reverse_kl_samples; mecka config enables M=4). Kept our wrap-aware cam MSE + *_nowrap companions. Train viz second head: TrainVizEvalVideo wraps the wristframe evaluator, runs against a train-split loader (dataloader_idx=1), metrics prefixed train_viz/, videos under videos_train_viz/, M-sample metrics forced off. val_at_start=true gives a pre-fit baseline val on fresh launches (skipped on requeue/ckpt-resume, where it would score un-restored weights). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wavefront DP over anti-diagonals, vectorized across the batch (2T-1 steps for a TxT grid). Tolerates temporal misalignment that paired MSE penalizes; logged as Valid/<pred>_dtw_avg on every val (train_viz inherits). Verified against a brute-force reference in tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tures 1,607 episodes (task='fold_clothes' == flagship collection exactly; freeform fold data is labeled 'folding_clothes' and excluded). val@10 epochs, viz@100 on both heads, norm stats reused from the full-mecka precompute. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ture crash) The compiled sampler's cudagraph recording does in-place setup on static buffers - illegal on inference tensors. Hit by the M-sample val metrics' repeated sample_actions calls (fold-debug 11112709, val_at_start). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18,456 eva_bimanual episodes (processed_v3/abc/). Wristframe-6D + 6D proprio (20D native, aligns with mecka's padded prompt layout). New init_weights_ckpt: weights-only Lightning-ckpt init with shape-mismatch filtering (fresh optimizer) - initialized from mecka epoch-849. All branch val metrics + train_viz head enabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ported from the remote repo (df7540a): abc eva ships 480x640 / 480x848 / 720x1280 episodes, which default_collate cannot stack. Bilinear-resize all camera tensors pre-collate; key match widened to cover PI-style *_rgb keys (our cartesian_pi keymap) alongside dataset-style 'images' keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pinch/curl heuristic (constants from the aria viz calibration) -> per-arm grip channel inserted at robot slots 9/19: human actions+proprio become 20D, layout-identical to eva, packed by the Robot converter -> gripper channels get real supervision instead of zeros. Grip keypoints ride the keymap with the action horizon and are strided/interpolated exactly like the pose chunker. Grip revert unpads actions before the standard wristframe revert. NO left-wrist fix for aria: verified on-data raw aria already matches the post-fix mecka convention (sign check 50/50 frames). Chain scripts: sync (709 eps) -> norm -> 1xH200 smoke -> 4xH200 train from mecka epoch-849. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… copies) Several shared-mirror aria episodes were stale partial copies (old syncs; 'already present' skip kept them) -> decode-retry storm, ~21 s/step, 148GB log on the first finetune attempt. Fresh sync into agao81/ariaFoldZarrDatasets with per-episode decode verification gating the train job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- data/model/sbatch for the combined human+eva pretrain (1:1 mix per step via CombinedLoader, 32/key/GPU -> global 512 at 8xH200) - eva.py: EVA_EXTRINSICS registry (x5Dec13_2 rollout-rig calib + abc_fold_viz ABC top-camera solvePnP fit) with extrinsics_key plumbed through get_transform_list; Eva.INTRINSICS switched from the ARIA fallback (fx=266.5, ~1.6x overlay mis-scale) to the ABC RealSense K - abc data bakes abc_fold_viz into the cam-frame proprio (matches the remote runs; wristframe action targets are extrinsics-invariant); the legacy abc_eva_pi_6d config keeps x5Dec13_2 for its shipped checkpoints - norm_abc_eva_foldviz.sbatch recomputes eva stats in the new frame and regenerates the merged mecka_abc_6d stats
- _split_mse re-instantiated two torchmetrics MeanSquaredError objects per call (twice per embodiment per val batch) and the shared `mse` was a stateful accumulator; both replaced by a stateless _paired_mse. - New Valid/<emb>_<key>_rot_geodesic_avg (native output, after the same Gram-Schmidt the decode uses) and _cam_rot_geodesic_avg (cam frame): arccos((tr(Rp^T Rg) - 1) / 2), computed in float64. The wrap-aware ypr MSE is still degenerate at pitch ~ +-pi/2, where yaw and roll trade off and a 0.004 rad orientation error scores > 0.1; the geodesic error is immune to both the +-pi wrap and that degeneracy. Tests cover both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aidang3019
force-pushed
the
aidan/eval_gating
branch
from
August 29, 2026 01:05
db5533a to
6c3913f
Compare
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.