feat(abc): ABC-130k mcap to zarr pipeline + yam_bimanual embodiment

[AnikethCheluva]

feat(abc): ABC-130k mcap to zarr pipeline + yam_bimanual embodiment

Adds egomimic/scripts/abc_process to pull ABC-130k (XDOF/ABC-130k) off the
Hub and convert its YAM MCAP episodes into EgoVerse zarr v3.

• download_abc.py: scoped non-recursive tree listing + per-file
  hf_hub_download. snapshot_download cannot be used here, it expands
  allow_patterns against the repo siblings list which the Hub returns
  empty for a repo this size, so every pattern matched nothing.
• inspect_abc_mcap.py: dump topics/schemas before converting.
• abc_to_zarr.py: decodes CompressedVideo per stream (codec read off the
  format field, h264 or h265), auto-detects mono RealSense vs stereo
  ZED-X stations, nearest-neighbour matches the few-hundred-Hz arm and
  gripper streams onto front camera frame timestamps, pulls real
  intrinsics from the sibling -info CameraCalibration, and maps
  subtask-annotation messages to frame ranges. JPEG-encodes during decode
  so a 5k frame 3 camera episode stays at 1.5G RSS instead of OOMing.

Registers YAM_BIMANUAL = 7 plus a Yam embodiment class in both class
registries, so viz resolves it instead of silently falling back to Human.

Yam.get_transform_list gives the wrist-frame pipeline, mirroring Eva's
eef-frame builder with the world to camera hop dropped. ABC records no
extrinsics but the wrist-frame path does not need them: actions are a
delta relative to the current EEF pose, so a rigid frame change applied
to both operands cancels, (E^-1 Tobs)^-1 (E^-1 Tcmd) == Tobs^-1 Tcmd.
Verified on a real episode, actions match to 7e-16 under an arbitrary
rigid E. Proprio does differ: Eva feeds a camera-frame EEF pose, YAM
feeds a station-world-frame one. There is no camera-frame mode for Yam.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

feat(abc): yam viz notebook section + tolerate episodes with no EE pose

Appends an ABC-130k/YAM section to zarr_data_viz.ipynb pointed at a
converted fold_and_stack_the_t_shirts episode in
/coc/flash7/scratch/acheluva3/abc_data. Loads it via LocalEpisodeResolver
(filters on zarr attrs, so no DB row needed), plays the top and wrist
cameras, lists the subtask annotations, and plots world-frame EEF paths
plus the wrist-frame action deltas and gripper.

No image-space trajectory overlay: viz_transformed_batch(mode="traj")
projects actions through K assuming camera-frame points, and wrist-frame
deltas sit near the origin, so it draws nothing (checked, 0 pixels
changed). That needs a world to camera transform ABC does not record.
Also note viz needs the 14D ypr mode, _split_action_pose rejects 16D.

abc_to_zarr: RobotState.pose is documented as always present but a real
slice of ABC ships it empty on every message of an episode (joints only,
2 of 6 sampled t-shirt episodes). Skip the pose key with a warning
instead of crashing on reshape, and fail with a clear message if no arm
stream has poses, since the cartesian pipeline needs them.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

feat(abc): yam revert transform + action-overlay video; fix stereo top camera

Yam.get_revert_transform_list mirrors Eva's revert builder: undoes the
wrist-frame step (inverse=False, re-composes the chunk onto the obs pose)
to recover absolute poses. Verified it reproduces the commanded poses the
converter wrote, exactly.

Adds an action-overlay video to the notebook. It is NOT drawn on the image
the way the Eva/Human sections do it. _viz_traj projects through K, which
only means anything for camera-frame poses; Eva gets that from
Eva.EXTRINSICS and ABC records none, so the revert lands in the station
world frame with nothing to project with. Confirmed no source has one: the
MCAP -info messages carry K/resolution/distortion only, the YAM MJCF
defines no cameras, and ABC's own export_mcap.py trains on joint space
while viz_policy.py is a 3D Viser view of a sim rollout. Fitting the camera
pose by PnP against motion centroids was tried and rejected, cloth motion
dominates the centroids and the reprojected grippers visibly miss. So the
chunk is drawn in a station top-down panel beside the frame, with the
subtask label and gripper state. Setting Yam.EXTRINSICS re-enables the
ordinary projected path.

abc_to_zarr: pick the stereo eyes before /top-camera. On a ZED-X episode
/top-camera also exists but is a 640x400 preview with the left/right insets
baked in, so the old order would have trained on a composited thumbnail.
Matches the rule in the dataset's own export_mcap.py. Mono RealSense
episodes are unaffected.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

feat(abc): recover top-camera extrinsics, real action overlay, frame-matched subset builder

The overlay gap is closed. ABC's MCAP has no extrinsics (checked: the
-info CameraCalibration messages carry K/resolution/distortion, R is
absent and P is K with a zero column; no /tf, no attachments; the
metadata record is camera specs only), but the rig itself is published.
i2rt-robotics/i2rt robot_models/station/yam_station_{crank,linear}_4310_d405
carries the chain, and composing top_camera_bracket -> top_camera_body ->
top_camera reproduces its README's documented extrinsics exactly,
including the stated check that the optical axis meets the base plane at
(0.384,-0.305,0) at 60 degrees. ABC's world frame is that model's
left_base: the documented 0.61m arm separation matches the data, and
projecting ABC EE poses through it lands them on the grippers.

• Yam.EXTRINSICS, alongside Eva.EXTRINSICS in form and place. D405
  stations only; ZED-X has no published rig, so abc_to_zarr attaches it
  only when the episode metadata reports a D405 top camera.
• Yam camera-frame pipeline (cartesian / cartesian_ypr) mirroring Eva's,
  both arms referred to the single overhead camera. viz_transformed_batch
  mode="traj" now works exactly as it does for Eva.
• Yam.viz_wristframe_batch: overlays the wrist-frame training
  representation by reverting onto the EEF pose and mapping to the camera
  frame, then draws the subtask label over it. The base viz modes are
  mutually exclusive so the annotation is a second pass, not instead of.
• build_abc_subset.py + scripts/abc_build_subset.sbatch: download,
  convert and drop raw MCAP until a FRAME budget is met. Budget rather
  than episode count because ABC episode lengths vary ~3x and about a
  third of t-shirt episodes ship no EE pose. Defaults to 316068 frames,
  the mecka fold_clothes total. Incremental, so a requeue resumes.

Notebook ABC section reformatted to match the Eva section, no striding
so the video plays at capture rate.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

fix(abc): shrink the in-flight window as the frame budget fills

A running future cannot be cancelled, so every episode in flight when the
budget is met still downloads and converts. With workers*2 held open that
overshot badly: a 3000-frame smoke test finished at 14146 frames, 471%.

Size the window from the frames still needed instead, using the mean
episode length seen so far (falling back to the 2861-frame task average
from meta/train_report.txt) with 1.5x headroom for the episodes that fail
to convert. Full width early, tapering to near-exact at the end.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

fix(abc): refine top-camera extrinsics to the real mount; repair viz notebook

Extrinsics: the published station model gives design intent, not per-station
calibration, and the bracket is bolted to a gantry crossbar. Projecting with
the nominal translation put the EE ~18px left of the gripper, consistently
for both arms and at every arm pose, which is the signature of a small rigid
mount offset rather than a pose error. Fit y and z against two hand-marked
gripper positions: +0.0413m and +0.0133m, 4.3cm total. Residual drops from
~18px to ~4px, and checked across six frames spanning very different arm
poses so it is not overfitting the one frame. Rotation untouched, it still
reproduces the published values exactly.

Notebook: the ABC section had lost its dataset-setup cell and the annotations
header, so everything downstream NameError'd on abc_ds. Restored, and the
section is now self-contained: it previously used MultiDataset, DatasetFilter,
torch, mpy, np and zarr while importing only Yam and LocalEpisodeResolver,
inheriting the rest from the Eva section 20 cells up, so running the ABC
section on its own failed. Verified by executing it in a fresh namespace with
no Eva/Human cells. Also hoists store/T/FPS into setup, removing a hidden
ordering dependency where the video cells needed the annotations cell run
first, and caps both videos at PREVIEW_SECONDS=30 (900 frames) at the capture
rate instead of rendering the full ~165s episode.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

feat(abc): backfill embodiment/intrinsics/extrinsics onto the R2 ABC episodes

The lab='abc' ingest wrote no camera metadata at all: zarr.attrs carries only
embodiment, features, fps, task_name, task_description and total_frames.
So the overlays cannot project on any of those 18456 episodes, and every one
is labelled eva_bimanual although the rig is a YAM station.

Intrinsics are per-episode and are NOT recoverable from the zarr, so they are
read back from the source MCAP on the Hub. Measured on a real episode, the
-info CameraCalibration messages sit in the FIRST chunk (byte 71) while
the metadata record is at 65MB, so a ~1.2MB ranged GET replaces a 66MB
download, a ~55x saving at ~1.1s per episode. The calibration's width/height
also identifies the station, 640x480 RealSense vs 1920x1200 ZED-X, which is
what decides whether the published D405 extrinsics may be applied at all;
ZED-X episodes get intrinsics only and are tagged in the manifest.

Parsing uses the raw StreamReader on the prefix. NonSeekingReader is not
usable here: its iter_messages sorts the whole stream before yielding, so on a
truncated prefix it only ever raises EndOfFile.

Reading K per episode rather than assuming a nominal is load-bearing, the
sampled fx spans 431.9 to 436.3.

Defaults to a dry run; --apply commits. The manifest records each episode's
previous embodiment/intrinsics/extrinsics so a run is reversible, plus
per-episode station and status. Threads, not processes: this is network bound.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

fix(abc): gate the D405 extrinsics on optics, not topic or resolution

Dry run over the fold-and-stack family (3143 episodes) showed /top-camera
carries two different cameras, not one. Normalised focal length splits into
two clusters with a wide empty gap between them:

fx/W ~ 0.678 2724 eps ~70 deg HFOV D405
fx/W ~ 0.34-0.51 419 eps much wider something else

Different camera hardware means a different bracket, so the published D405
pose must not be applied to the wide ones. Neither earlier gate caught this:
resolution rejected legitimate D405 capture modes (848x480, 1280x720 share the
same mount, only K differs and K is rescaled per episode), and topic accepted
every mono camera regardless of optics. Gate on fx/width, which is invariant
to capture mode and is the thing that actually identifies the lens.

The wide-angle episodes now get intrinsics only and are tagged
wide_angle_unknown_rig(fx/W=...) in the manifest so they can be revisited if
a rig model for them turns up.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

feat(abc): upload locally converted episodes to R2 and register them in SQL

fold_and_stack_the_t_shirts is absent from the lab='abc' ingest entirely (110
of the Hub's 201 train tasks were ingested, t-shirt folding is one of the 91
missing), so the 114 locally converted episodes are net-new rather than a
re-upload.

Per episode: refresh zarr.json extrinsics to the CURRENT Yam.EXTRINSICS so what
lands in R2 carries the refined mount transform rather than the stale nominal
one baked in at conversion time; upload the store under
processed_v3/abc/.zarr/; insert one app.episodes row.

rig_name is 'abc' and embodiment yam_bimanual, matching what the rest of the
lab='abc' rows now carry. TableRow has no license field so that column is set
afterwards to match the ingest.

Zarr sharding keeps a store to ~35 objects (3981 files / 38GB across all 114),
so this is multipart PUTs rather than a small-file storm. Upload and SQL insert
are separately idempotent, so a partial or requeued run resumes.

Defaults to a dry run; --apply commits.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

fix(abc): correct K from the principal point, gate extrinsics on FOV

Two defects found by projecting the EE with candidate K's and looking at which
one lands on the gripper.

1. ABC's CameraCalibration is internally inconsistent on ~30% of episodes: the
   width field reads 640 while K is still the one for the 848 wide capture it
   was resized from (cx ~ 425 = 848/2, fx unscaled). Trusting width left the
   principal point ~105px off, and the overlay missed the gripper by about
   that much. Derive the true extent from the principal point instead: a
   calibrated centre sits near the middle, so 2*c is the extent K belongs to.
   cx/W is sharply bimodal (~0.49 vs ~0.66, nothing between), so the override
   only fires where the declared size clearly disagrees.

2. fx/width does not identify the lens. The same D405 appears in two capture
   modes -- 640x480 as a centre crop (fx/W0.68) and 848x480 resized to 640
   (fx/W0.51) -- with very different fx and an identical mount. The old gate
   split that second group across both sides of the line depending on whether
   the cx bug had shifted fx. Gate on horizontal FOV computed from the
   corrected K, which is invariant to capture mode and resize and separates
   with a wide empty gap:
   72-73 deg D405 640x480 crop
   88-89 deg D405 848x480 full frame (~87 deg native, matches the part)
   96-97 deg a genuinely wider camera, no published rig

The manifest now records cx as well as fx so this is auditable.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

chore(abc): backfill sbatch accepts ALL and runs on 16 cpus

The task-substring selector could not express "every lab='abc' episode",
which is the run needed to cover the ~15300 episodes outside the
fold-and-stack family. Bumped to 16 cpus since that is ~6x the work.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

fix(abc): back off on HTTP 429 and cut backfill concurrency

The first full pass over all 18570 lab='abc' episodes lost 10248 of them to
HTTP 429 from the Hub. There was no retry, so a rate-limited episode was simply
recorded as failed.

Retry ranged GETs on 429, honouring Retry-After when present and otherwise
backing off exponentially with jitter so workers do not resynchronise into
another burst. Drop the default worker count from cpu*2 (32 on the backfill
node) to at most 12: the ceiling here is the Hub's rate limit, not local CPU,
and the run is only ~1.2MB per episode.

The pass is idempotent, so a re-run skips the 8322 already written and retries
only the episodes that failed.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

feat(yam): match Eva cartesian 14D layout and add cartesian_world

Eva cartesian is 14D YPR; Yam had is_quat inverted and emitted 16D.
Follow Eva's convention (convert to YPR at the end) and add cartesian_world
plus dinov3_keymap so Yam can drop into Eva config slots. Arc tokenizer
stays off this branch.

feat(6d): add cartesian_wrist_6D Zhou rotation for Eva, Yam, Human

Wrist-frame actions keep xyz and gripper, replace ypr with the first two
columns of the rotation matrix (9+1 per arm = 20D). Human pads a zero
gripper so it shares that layout.

feat(yam): Eva parity on transforms, keymaps and action layout

Yam now matches the Eva surface that exists on this branch, so it can drop
into any config slot Eva occupies: cartesian, cartesian_world, both
wristframe modes, and a dinov3_keymap identical to Eva's.

Fixes an action-layout divergence. Eva's is_quat describes the INPUT (the
converter writes xyzwxyz) and its pipeline converts to YPR at the end via
XYZWXYZ_to_XYZYPR, so Eva "cartesian" emits 14D. Yam had the flag inverted and
emitted 16D. Yam now follows Eva verbatim; verified the two YPR converters
agree to 4e-16, so this is a shape and semantics fix, not a numerical one.
cartesian_ypr stays as an alias since the viz notebook names it.

cartesian_world differs from Eva in meaning, not shape, and says so: Eva's base
IS its front camera so its world mode stays projectable, while YAM's top camera
sits ~0.95m above the base at 60 degrees, so world-frame poses need EXTRINSICS
before they can be drawn on the image.

arc_tokenizer_cartesian is deliberately NOT here: it imports
rldb/zarr/arc_length_tokenizer, which lives on the arc branch, and Eva has no
arc mode on this branch either. It belongs with the rest of the arc-tok work.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FqThupNm4UFnd2xvKDYnAM

feat(abc): resize images to a common size and rescale K to match

A batch cannot mix image sizes and this data does: ABC ships 640x480, 1280x720
and 848x480, mecka ships 640x360 and 640x480. Resolvers take image_hw and every
episode is resized to it, with that episode's intrinsics rescaled by the same
per-axis factors so projected overlays stay exact.

backfill_abc_metadata was scaling fx alone when the declared width disagreed
with the principal point, producing an anamorphic K (fx=325 next to fy=432) on
21 episodes and squashing their overlays 25% in x. The 848->640 case keeps its
height, so it is a crop: focal length is unchanged and only cx shifts. Resize
and crop are now told apart by aspect.

Human revert gains gripper_padded so the 14D [L pose, L grip, R pose, R grip]
layout splits correctly instead of reading L-grip plus five columns of R-pose
as the right arm.

Also registers the yam viz func so yam validation videos render.

[AnikethCheluva]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Aniketh/arc #573
• feat(abc): ABC-130k mcap to zarr pipeline + yam_bimanual embodiment #606 👈 (View in Graphite)
• feat(val): nested validation dataset groups #605
• refactoring aciton transforms #604
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Claude Code Review

Review of PR #606: ABC-130k pipeline + yam_bimanual embodiment

Summary

Adds a full ABC-130k ingestion pipeline (MCAP → Zarr v3), a new YAM_BIMANUAL embodiment mirroring Eva's transform surface, plus backfill/upload utilities and cotrain configs. Extensive rigor in the description — geometry, extrinsics fitting, and edge cases are all well-justified.

Key concerns

1. Optimizer/scheduler confound in the BC baseline (hpt_abc_bc_flow_baseline.yaml). You explicitly flag this yourself in the config comments: baseline inherits lr=1e-4 + CosineAnnealingLR from bc_flow_eva, while the cotrain runs use lr=3e-4 constant. If this pair is the intended controlled comparison for the cotrain ablation, the run isn't clean. Either uncomment the override or add a note in the PR that this pair is not the ablation control (the arc twin is).

2. Yam.EXTRINSICS is a mutable class-level dict initialized to the D405 pose, but the docstring and abc_to_zarr say ZED-X / wide-angle episodes have no valid extrinsics. Anything that reads Yam.EXTRINSICS["front_1"] at transform-build time (the camframe pipeline, viz_wristframe_batch with to_camera_frame=True) will silently apply the D405 pose to a ZED-X episode if a config picks coord_frame=camframe on mixed data. There's no runtime check that the episode's station matches. Consider gating on zarr.attrs["station"] or per-episode extrinsics from attrs, matching how intrinsics are handled.

3. _flatten_annotations mutation bug. stack = list(item) + stack inside a while stack: item = stack.pop(0) loop is O(n²) and, more importantly, mixes prepend semantics with pop-from-front — the traversal order is not what a reader expects. Minor, but likely to bite. Prefer a proper recursive flatten or collections.deque.

4. Backwards-compat break in _build_human_cartesian_revert_eef_frame_transform_list. New gripper_padded kwarg defaults to False, but the accompanying commit says Human 14D "action_mode=arc_tokenizer_cartesian_gripper_padded" needs padded=True. Anywhere existing configs use 14D padded human data through this revert path (viz), the split will silently misalign L-grip into R-pose. Please confirm all call sites are audited — a grep of _build_human_cartesian_revert_eef_frame_transform_list and any Human revert wrapper is warranted.

5. HPT split_6dof 20D case. The unpack assumes layout [L xyz(3) rot6d(6) g(1), R xyz(3) rot6d(6) g(1)] = 10+10. Please confirm this exactly matches cartesian_wrist_6D transform output ordering — the description says the wrist-frame transform produces "9+1 per arm" but nothing in the diff shows the concat order. A unit test that round-trips an action through get_transform_list(...) and split_6dof(...) would catch layout drift.

6. Test coverage. For a change of this scope (new embodiment, new coordinate math, new backfill that mutates R2 zarr.attrs, new upload that writes SQL rows), I see no unit tests in the diff. At minimum I'd want:

   • Round-trip: get_transform_list(eef_frame) → get_revert_transform_list() recovers commanded poses to numerical tol (you say you verified this manually — please codify).
   • _scale_intrinsics resize-vs-crop branch on the 848→640 case.
   • backfill_abc_metadata dry-run correctness against a synthetic MCAP prefix.

7. backfill_abc_metadata writes to R2 with --apply. This mutates the shared processed store used by all lab='abc' training. The manifest-based reversibility is good, but the PR description shows the extrinsics have been refined twice during this PR's lifetime (nominal → +18px offset fit → FOV gate). Please confirm the final apply happened after all refinements landed and the manifest reflects the current Yam.EXTRINSICS, not a stale one.

8. operator SHA-256 hashing. The new SQL insert path in the upload script isn't shown in the truncated diff. Standard convention reminder: any TableRow construction must hash operator before insert.

Suggestions

• Add Yam.EXTRINSICS per-episode lookup (or a runtime assert that the episode station matches the class-level pose). Currently the class-level dict is a footgun for ZED-X + wide-angle mixed batches.
• In hpt_abc_bc_flow_baseline.yaml, either commit to the override or rename to make the confound explicit (e.g., _baseline_eva_optim.yaml).
• Fix _flatten_annotations traversal to be a proper recursive/deque flatten.
• Add round-trip and split_6dof unit tests before this merges — the geometry is load-bearing for every downstream policy trained on this data.
• Confirm the ~10248 episodes that hit HTTP 429 in the first pass have all been re-run to success and the R2 store is now complete for lab='abc', or scope this PR to explicitly state the coverage delta.
• The truncated diff hides the upload script, _scale_intrinsics, and backfill_abc_metadata fully — I'm flagging these as needing individual review; they touch R2 and SQL and should not go in on a rubber-stamp.

Verdict: Request Changes

The core work is careful and well-documented. Blocking items are (a) the optimizer confound in the BC baseline if that's meant to be an ablation control, (b) the extrin

---

Reviewed by Claude · Review workflow