Aniketh/arc - #573
Open
AnikethCheluva wants to merge 90 commits into
Open
Conversation
AnikethCheluva
force-pushed
the
aniketh/arc
branch
from
August 26, 2026 14:30
a412dcf to
6617e22
Compare
Claude Code ReviewReview of PR #573: Aniketh/arcSummaryThis PR is a massive, unfocused omnibus change touching simulator versioning ( Key concerns1. Scope / reviewability (blocker)The diff was truncated at 80k chars — I can't see the full change. What is visible spans at least 6 largely-independent efforts:
Each of these deserves its own PR. As-is, this is unreviewable and unmergeable. 2. BATCHFLOW.md contradicts what the code actually does
Also: the doc references repo 3. Convention violations
4. Data-integrity risks in the packed pathFrom what I can see:
5. Training regressions
6. Test coverageThe
7. Minor
Reviewed by Claude · Review workflow |
The reconciliation layer. Every file where main and the batchflow lineage genuinely disagreed lands here, so everything above it is bulk-new code. embodiment.py takes main's collapsed enum (HUMAN_* 1-3, EVA_* 4-6) and re-adds PUSHSHAPES_SIM 15 / _STICK 16 / _SMALL_CIRCLE 17 -- pinned because trained checkpoints and collected datasets encode those IDs. zarr_dataset_multi.py is a 3-way merge against the fork point: main's SafeS3EpisodeResolver, EvenStrideDataset, _evenly_spaced_indices and intrinsics property, plus batchflow's _read_span, _annotations_for_span and LocalEpisodeResolverWithEmbodimentOverride. action_chunk_transforms.py keeps batchflow's DeltaAction alongside main's PadGripperZeros. The batchflow repo notes land in AGENTS.md rather than a second CLAUDE.md, so the repo keeps one conventions file. DESIGN.md is not carried over: it was a 2026-06-06 restructure proposal written against EgoVerse-pact-2, still marked "awaiting approval", describing a move that has since happened here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The JPEG-decodability probe at _probe_image_key calls simplejpeg.decode_jpeg(...) but the module was never imported in this file. The call sits inside a try/except Exception, so instead of crashing it made the probe report EVERY image as undecodable -- a silent false negative rather than an error. _common.py in the same package already imports it the same way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AGENTS.md still documented egomimic/models/hnet_nets/, egomimic/algo/hnet.py and egomimic/eval/eval_hnet.py. None of those paths exist: the packages are models/hnet/, algo/hnet/ and eval/core/eval_hnet.py. Anyone -- human or agent -- following the doc went looking for files that are not there, and AGENTS.md is the first thing an agent reads. Section headings renamed hnet_nets -> hnet to match. test_hnet_nets.py is left alone: that file genuinely still has that name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oader
This PR introduces the packed subsystem -- ZarrEpisodePackedDataset and
pack_collate -- but MultiDataModuleWrapper, which is inherited unchanged from
main, hardcoded annotation_collate for every dataset. annotation_collate ends in
default_collate, which tries to torch.stack ragged packed samples, so every
packed_episode config died on its first batch with
RuntimeError: Trying to resize storage that is not resizable
pack_collate needs two call sites. The other one -- MultiDataset's norm-stat
inference in zarr_dataset_multi.py -- already had it (collate_fn = pack_collate
if is_packed else None) and is covered by test_packed_pipeline. The training
dataloader had neither the wiring nor a test, and main has no packed configs at
all, so nothing exercised it.
_collate_fn_for is ported from EgoVerse-gmm-dualstream / EgoVerse2, where this
dispatch already backs the live H-Net runs, rather than written fresh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two standalone simulator packages, each with its own pushshapes/, collect/, examples/ and tests/. There is no v3: what was labelled v3 is the socket fix that ships AS v2 -- the intermediate all-faces-grip build was a bug. __init__.py aliases the active version's submodules to the top level so existing 'from Tsimulation.pushshapes import X' call sites keep working; TSIM_VERSION selects the version per process. Placed before eval because eval calls into it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hing on them
Adding the u_socket was done by editing the environment. Its latch, friction and
penetration guards -- 12 methods, ~470 lines -- went into env.py as
`if pusher_shape == "u_socket"` branches, and its 3-DOF action became a hardcoded
`expected_shape = (3,) if self.pusher_shape == "u_socket" else (2,)`. That is the
single largest reason the two sims diverged: sim_v1's env.py has ZERO socket
references, sim_v2's had 115, and env.py's step loop called nine socket-specific
guards in sequence.
An Agent now owns the three things the environment should not know about:
* ACTION SPACE -- action_dim (2 for a free-moving pusher, 3 when the agent
also controls orientation) and target_pose() to decode a raw action;
* BODY -- build() in the pymunk space;
* CONTACT MODEL -- pre_substep()/post_substep() hooks around each physics
substep, plus on_reset() for per-episode state.
env.step is agent-agnostic:
captured = self.agent.pre_substep(self)
self._drive_pusher_toward(tx, ty, dt_sub, target_angle)
self._space.step(dt_sub)
self._clamp_pusher_to_static()
self.agent.post_substep(self, captured)
Agent (circle, circle_small, stick, L) implements the hooks as no-ops.
USocketAgent owns all the latch/guard logic and the socket geometry constants,
and its solid_pusher / socket_inside_friction_only flags become constructor
arguments rather than environment state. A new agent with an unusual action
space is a new class plus one line in make_agent(), not another branch in the
simulator.
env.py 1292 -> 795 lines.
sim_v1 IS DELIBERATELY UNTOUCHED. It is frozen so pre-rewrite data replays
exactly; refactoring it would put that at risk for no benefit, since it has no
socket to abstract in the first place.
VERIFIED BY REPLAY EQUIVALENCE, not by inspection. Baselined the unmodified sim
with the identical harness first, then compared:
u_socket_3000_v2 100.0% -> 100.0% (p50 0.0033 -> 0.0033)
circle_3000_plus_gen_v2 89.7% -> 89.7%
circle_v2_obstonly 17.1% -> 25.7%
The gate caught two real bugs that inspection did not: the moved
_socket_contact_is_on_inner_face call site lost its env argument, and
socket_latched -- a property DERIVED from `_socket_constraints is not None` --
had become a plain attribute nothing updated, initialised to [] so it would have
read as permanently latched. Both fixed; the socket went 0% -> 100%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012V58H37tmcvgDthELMd5Xk
New subsystems with no counterpart on main: models/hnet (stages, blocks, the scan and register chunk interfaces, routing) and models/diffusion (DiT3D and spatial backbones, sampling, image VAE), plus cores and stems. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
widths=[512] in hpt_heads/hpt_stems and down_dims=[256,512,1024] in denoising_nets are evaluated once at import, so every caller that omits the argument shares one list object. None of the three currently mutates it, so nothing is broken today -- this removes the footgun before something does. Each becomes None with the original value restored inside the function, so the behaviour for an omitted argument is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
obs_encoder.py and cond_encoders.py each carried a private _mlp() of the same shape -- but ObsEncoder's used ReLU and CondEncoderModule's used GELU. Both are live: ObsEncoder backs nine bc_rnn model configs (base/tx/tx_chunk8/hnet/ tx_cotrain_*), CondEncoderModule backs the H-Net firstend/cotrain/bf_rh configs. They collapse into one build_mlp() whose 'act' argument is keyword-only with no default, so every call site states its activation. That is the point of the change: activations hold no parameters, so a checkpoint trained under one activation loads into the other with no error and merely produces different numbers -- exactly the kind of silent divergence two near-identical private helpers invite. Behaviour is unchanged: ReLU stays at both ObsEncoder call sites and GELU at both CondEncoderModule ones. Verified by building both encoders from the old and new trees -- identical activation lists, identical parameter keys, identical forward output hashes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…thms algo/hnet, algo/diffusion (with its outer-stage variants) and algo/bc. Builds on the model zoo in the previous commit. Also carries egomimic/pipeline/ -- the batchflow stage framework, its runner PipelineAlgo and the stage implementations -- plus the three H-Net lightning callbacks (random_attn_dropout, chunker_residual_scheduler, ratio_loss_scheduler) and BATCHFLOW.md. These sit here rather than in the infra commit because they import egomimic.models.hnet and egomimic.models.diffusion from the model zoo below, and pipeline/algo.py additionally imports egomimic.algo.hnet.episode_transforms from this commit. Carrying them lower left the infra commit unable to import six of its own modules when checked out on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HNetPolicy.step referenced embodiment_id at two places in its body but never
declared it -- not a parameter, local, or attribute. The method failed two ways:
* PackedAlgoBase.inference_step already calls
policy.step(..., embodiment_id=self.domain_by_id.get(emb_id)), which raised
TypeError: unexpected keyword argument;
* called without it, the body raised NameError at the action_out lookup.
Either way the AR single-step path used for closed-loop sim rollout could not
run. The sibling policy step() in this same file already declares
embodiment_id: Optional[str] = None; this matches it, so the existing callers
work unchanged and single-embodiment models keep the None default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
auxiliary_ac_keys: dict = {} and aux_ac_keys=[] are evaluated once at import and
shared by every caller that omits them. Neither is mutated today (the dict is
copied on assignment, the list is only iterated), so this is prevention rather
than a live bug fix.
Both become None and are materialised inside the function, leaving the
omitted-argument behaviour identical. Adds the missing typing.Optional import.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ages DualStreamChunkedOuterStage and MultiStreamOuterStage both already subclass DualStreamOuterStage, and all three opened encode() with byte-identical copies of the same ~27 lines: the packed-only guard, unpacking actions/__obs, deriving T_total/device/dtype, casting cu_seqlens, accumulating the SPECIFIC stream over input_modules, and computing the AGNOSTIC stream. That block is packed-sequence boundary handling. A cu_seqlens device or dtype fix applied to one copy silently missed the other two -- three places to get the same thing wrong, with no test that would notice. Three helpers on the shared base replace it: _packed_inputs (guard + unpack, taking the caller name so each class keeps its own NotImplementedError message), _specific_stream and _agnostic_stream. Control flow in every encode() is otherwise untouched; only the derivation moved. Net -37 lines. Verified: ruff F401/F821/F841 clean on both files; all three classes import and inherit the helpers; the packed-only guard still fires per class with its own message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
egomimic/algo/diffusion/ (the DFoT algo and its nine outer stages) moves to its own PR stacked on top of this one. Nothing on main depends on it, and nothing remaining in this stack imports it -- models/diffusion stays, because the batchflow pipeline imports SinusoidalPosEmb from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HNetPolicy.init_step_state and HNetOuterStage.init_step_state were the same 27
lines -- same signature, same T_max coercion, same seven-key state dict --
differing only in which submodule owns the KV cache (self.hnet vs
self.inner_stage). Neither class has a base to hang it on, so it becomes a
module-level _init_ar_state(module, cache_owner, ...).
Worth more than its line count because both copies carried
dtype = dtype or next(self.parameters()).dtype
Allocating this state at a fixed dtype instead of the model's own is what once
produced a bf16 rollout state under fp32 weights, under-measuring H-Net
closed-loop coverage ~2-2.6x and reading as the policy failing closed-loop. One
copy of that derivation is the right number.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eval/core, eval/dfot and the explorer. The sim landed earlier in this stack, so the lazy 'from Tsimulation.pushshapes import ...' call sites here resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
egomimic/eval/dfot/ and egomimic/eval/tf/ move to the DFoT PR stacked above. egomimic/eval/__init__.py loses their entries. That registry is not lazy despite its name -- it import_module()s every entry at package-import time, so 'import egomimic.eval' was pulling the whole DFoT tree in. eval/core/img_utils.py mentions the DFoT evaluators only in docstring :mod: cross-references, not imports, so core is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tests covering the preceding layers, plus the scripts worth keeping: the CUDA kernel build, ops/, the eval entrypoint, and the sim replay tooling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
They exercise egomimic.algo.diffusion / egomimic.eval.dfot, which move to the DFoT PR stacked above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No test referenced MultiDataModuleWrapper at all, which is why the missing pack_collate dispatch survived: test_packed_pipeline covers the *other* pack_collate call site (norm-stat inference), so the subsystem looked tested. Asserts the dispatch itself and that it reaches the DataLoader, for both the train and valid loaders, and that unpacked datasets still get annotation_collate. Uses MagicMock(spec=ZarrEpisodePackedDataset) -- the same pattern test_packed_pipeline already uses -- so it needs no dataset on disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All four groups were stale tests, not broken code. Nothing under egomimic/ is touched by this commit. test_training_recipe (7): PackedAlgoBase takes an assembled outer_stage now -- action_dim / action_horizon / d_model / cond_encoder / hnet moved onto HNetOuterStage. The tests still passed them flat, so __init__ raised TypeError for the missing positional. They now build the HNetOuterStage and pass it; what each test asserts (lr stamping, parameter_groups, init_weights_range) is unchanged. test_pi (4): skipped at module level, not repaired. They describe a PI.visualize_preds API that exists nowhere -- no branch of this repo defines it (main included), egomimic.algo.pi.algo exposes no draw_actions to monkeypatch, and neither EgoVerse2 nor EgoVerse-gmm-dualstream implements it, so it was never ported into this lineage rather than dropped from it. Kept rather than deleted so the intent stays on record. (Their first error was a stale aria_bimanual domain, renamed to human_bimanual by the human/eva collapse; fixing that only exposed the missing method underneath.) test_core_defaults_byte_identical (1): the tx forward checksum was never a valid invariant. tx's output cancels from an absolute sum of 7163 down to a signed sum of ~1e-5, so the value moves with BLAS reduction order -- measured -1.21e-05 at one thread vs -4.43e-05 at two, on identical weights -- while the test compared it to a fixed ~1e-6 absolute bound. The reference simply captured one machine's rounding noise; refreshing it would fail again elsewhere. The guard is now the ABSOLUTE sum, stable to ~1e-8 relative across thread counts and still sensitive to any real forward change, with the signed sum kept at a tolerance scaled to the magnitude summed. lstm and hnet were unaffected because their sums are O(1) and O(100) and do not cancel. test_packed_pipeline (1): PUSHT_FOLDER pointed one directory level too high. circle/ now holds a basic/ subfolder rather than .zarr episodes and the resolver does not recurse, so the isdir() skip-guard passed while the resolver matched nothing. Same drift as data/pushshapes/packed_episode/simulation/delta.yaml. Correcting it also un-skipped three dataset-gated tests, which pass. Suite: 13 failed / 297 passed -> 0 failed / 309 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
data, experiment, evaluator and callbacks groups, plus every model config the
rest of the tree actually references.
An earlier pass kept 3 model configs. Scanning hydra_configs, tests and scripts
for names that resolve against the model group finds 62 referenced, so that
prune broke two things at once:
* all 31 experiment configs -- each carries an `override /model:`, so hydra
fails at `Could not find 'model/<name>'` before any code runs;
* 25 cases in tests/test_config_compose.py, covering the dfot_*, vae_* and
bc_rnn_pushshapes_paperexact_* families.
The 59 missing configs are restored here, bringing the model group to the 62
that are reachable from the tree. Two of them (bf_rh_sdp_dual,
bf_rh_sdp_nodual) are the only configs that instantiate egomimic/pipeline, so
without them the batchflow runner shipped unreachable.
The remaining 181 per-arm variants on elmo/batchflow-core are still
deliberately excluded -- nothing in this tree references them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…amilies
The model group was 62 flat files, 9,581 lines, with exactly one config
(pi0.5_bc_eva) using defaults-inheritance. Everything else was authored by
copying a whole file and editing a few numbers.
Folders. Each model family gets its own directory and the filename drops the
prefix the folder already carries:
hnet_cotrain_cossim_s4_trunk_200M.yaml
-> hnet_cotrain_cossim/s4.yaml (model=hnet_cotrain_cossim/s4)
64 configs, 10 folders, none left at the root. Every reference was rewritten:
8 in-repo files plus 26 scratch launchers outside the repo. Old -> new mapping
is saved at scratch/name_mapping_full.txt, since model names appear in the
Results Ledger and run pages as provenance.
Deduplication. Two families were literal copies differing only in numbers, and
both are now base + thin variants:
* hnet_cotrain_cossim -- 12 configs, 270 of 304 lines identical across all of
them (89%). The whole family is one 5-stage skeleton with a different split
of transformer depth between the per-embodiment levels and the shared apex.
Now 8 knobs; each variant is ~13 lines. 3,642 -> 961 lines.
* hnet_dualstream_txar -- 3 configs, 9 knobs. txar_m16 turns out to differ
from txar in k_agnostic/k_specific alone (3/2 -> 10/6).
Model group overall: 9,581 -> 6,414 lines.
The knobs are top-level scalars referenced by interpolation rather than
overridden in place, because outer_stage.hnet.stages is a YAML *list* and
OmegaConf replaces lists wholesale on merge -- a variant cannot override one
element of it.
Defaults entries need @_here_ (e.g. `- hnet_cotrain_cossim/base@_here_`).
Without it Hydra derives the package from the folder path and merges the base
under model.hnet_cotrain_cossim instead of the model root; the config still
composes and the tests still pass, it just silently inherits nothing.
The other seven families are NOT factored. Each has line-count differences
between members, i.e. they are structurally different models that share
boilerplate rather than copies of one template, so the same mechanical proof
does not apply. Duplication left: dfot 31%, bc_rnn 58%, vae 62%,
hnet_pushshapes 39%, hpt 34%, 2trunk 44%, bf_rh 39%.
Verified: parameterization proven lossless by reconstruction (11/11 cossim,
3/3 txar reproduce the originals byte-for-byte) before anything was written;
all 31 experiment configs compose; tests/test_config_compose.py 25 passed; and
the resolved model config of every one of the 62 pre-existing configs was
compared across the rename -- 48 identical, 14 differing only by the added
knob keys, 0 real differences.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
train_zarr_keypoints defaults pull in evaluator/viz/eva_cartesian_aria_keypoints_wrist, which does not exist in the repo. Not needed for a shape probe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
train_zarr_keypoints.yaml overrides evaluator/viz@evaluator.viz_func to 'eva_cartesian_aria_keypoints_wrist', which is not in the repo, so the config cannot compose. ~evaluator does not help -- the override lives in the defaults list and hydra still resolves it. Use keypoints_wrist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chunk is (100, 138) = 2 x (21*3 + 6): wrist is 6-dim xyz+ypr with no gripper slot, not the 7 I assumed, so the space is 138 not 140. Confirm which block is which by bone-length test -- correct slicing yields anatomically sized bones that stay rigid across the chunk; wrong slicing does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Probed the live pipeline rather than assuming: actions_keypoints is (100, 138) = 2 * (21*3 + 6). The wrist carries xyz+ypr only -- aria has no gripper and nothing pads one on this path -- so the space is 138, not the 140 I had, and the wrist block precedes the keypoints within each hand. Confirmed by bone-length test, which cannot be fooled: slicing as [Lwrist6|Lkp63|Rwrist6|Rkp63] yields 39.7mm bones with 0.38mm variation across a chunk, while keypoints-first yields 330mm "bones" with 3.7m outliers. The velocity slot moves to the wrist block accordingly. Re-verified end to end at D=0.45/M=30: token (31, 138), keypoint reconstruction 0.200mm (linf), and bones survive the round trip at 39.1mm with 0.61mm variation, so the hand stays rigid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds mode keypoints_wristframe_ypr_arctok, which runs the existing wrist-frame keypoint pipeline and then arc-tokenizes the (T, 138) chunk to (M+1, 138). dt = stride/30 for the same reason as the cartesian arc path. Two data configs on the SAME population as the cartesian runs (lab=rl2, task=fold_clothes), stride=1, with a real 20% holdout -- the existing aria_keypoints* configs set valid = train, so they cannot measure generalization. The pair differs only in parameterization: kp_human_baseline time-uniform, (100, 138) kp_human_arctok arc-uniform, (31, 138) at D=0.45 / M=30, linf Identical 138-dim action space, so this isolates tokenization rather than confounding it with action space. No model change needed for the baseline -- hpt_bc_keypoints_base is already single-domain with act_dim 138. The arctok variant only overrides act_seq to M+1 = 31. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow hardcoded data.{train,valid}_dataloader_params.eva_bimanual.*,
so any human-only config died with "Key 'eva_bimanual' is not in struct" --
after the dataset pull and norm stats had already run. Build the overrides
from an EMBODIMENTS list instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hpt_bc_keypoints_base sets lr=5e-5 with CosineAnnealingLR(T_max=1400). scheduler_interval defaults to "step", so T_max is 1400 STEPS ~ 14 epochs at the observed 100.8 steps/epoch -- and CosineAnnealingLR keeps evaluating cos(pi*T_cur/T_max) past T_max, so the LR climbs back up rather than stopping. Over 600 epochs that is ~22 sawtooth cycles between 1e-5 and 5e-5. Predicted 1.44e-05 at step 12299 vs 1.8e-05 observed on wandb. Every arc-cartesian run used constant lr 3e-4 with scheduler: null, so the keypoint runs as launched were neither constant nor at the same LR, and not comparable. Add const-LR variants. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
viz/keypoints.yaml and viz/keypoints_wrist.yaml both use a bare front_img_1 image_key, but the batch carries observations.images.front_img_1 (confirmed by probe) -- so neither can run. Add viz/keypoints_human.yaml with the correct key, no eva entry (these runs are single-domain), and no annotation_key, since arc_tests has no annotations. The eval sweep hardcoded annotation_key=null for four cartesian viz entries; on a keypoint evaluator those keys do not exist and hydra errors. Drive them from an ANNOT_EMBODIMENTS list instead, same fix as the dataloader overrides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same bug as the training workflow, fixed there but not here: the sweep set data.valid_dataloader_params.eva_bimanual.*, so any human-only run died with 'Key eva_bimanual is not in struct'. Drive from EMBODIMENTS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third instance of the same assumption in this file. Drive it from EMBODIMENTS like the others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Walks a real frame pair through the computation: two hand poses with per-joint displacement, the 21 distances as bars with L-inf/L2/L1-mean marked, the reduction to a scalar, the rotation term, and accumulation into tokens at D. Uses frames 2731->2737 where the index fingertip moves 37.9mm while the slowest joint moves 6.9mm, so the choice of norm is visible: L1-sum is 8.3x L-inf on that single step, which is what compounds into the ~22x path inflation over an episode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r_training zarr_key_to_keyname returns None for any batch key that isn't a registered action/proprio zarr key (intrinsics, episode_hash, image keys). The old 'if key is not None' guard checked the wrong variable — 'key' is a string from _batch.items() and never None, so the branch always fired and wrote every unregistered key under a single None dict slot. Later writes clobbered earlier ones, and 'intrinsics' was silently dropped. Downstream _intrinsics_from_batch(batch, i) then returned None, so Human.viz / Eva.viz fell back to the hardcoded class INTRINSICS constant. For episodes whose per-episode K disagrees with the aria default (mecka fx=fy≈251, cy≈184 vs ARIA cy=240), this projected the GT trajectory ~55px vertically offset — the visible wrist-vs-palm misalignment reported against arc_tests mecka fold_clothes val-videos. Fix: fall back to the original key when zarr_key_to_keyname is None so unregistered keys survive the rekey. Also fixes downstream access to episode_hash and per-episode image side-channel keys.
Layers on top of arc-length-nv-eval only what wasn't already there:
data configs (D=0.20m / M=15 hardcoded to match target convention):
- aria_train_mecka_val{,_arctok}: cross-domain (train aria fold, valid
mecka fold_clothes)
- eva_only_fold{,_arctok}: eva-only robot-baseline runs
- mecka_folding_clothes{,_arctok}: mecka folding_clothes (note "ing"
variant of task name — distinct from arc_tests fold_clothes)
- mecka_folding_eva_fold_cotrain{,_arctok}: cotrain across mismatched
task names (mecka folding_clothes + eva fold_clothes)
hydra launchers (mirror target's submitit_pace_l40s.yaml convention):
- submitit_pace_a100 / _blackwell / _h100
viz + diagnostic scripts:
- egomimic/visualization/arc_tok_viz.py: detokenize+overlay helper for
notebooks (mirrors ArcTokEvalVideo viz path)
- scripts/pixel_check.py + .sbatch: verify projected GT dots match
val-video mp4 frame 0
- scripts/visualize_trunk_latents.py + .sbatch: t-SNE + HDBSCAN over
HPTModel.forward_features, emits arc_embedding_sweep's tabbed HTML
small UI fix in scripts/arc_embedding_sweep.py: image-panel close
button now has type=button + inline onclick fallback.
Dropped from the pre-rebase stash because target already had them
(and more evolved): eval_arctok/eval_hpt/eval_video changes,
arc_tests_cotrain* config edits, D40_M100 model/evaluator configs.
Dropped as no-longer-wanted per user note: FMPolicyWithVelDecoder /
WithVelReadout wrappers, associated hpt.py / denoising_policy.py /
hpt_nets.py / model config _veldec / _velreadout variants. Also
dropped an accidental LocalFolderEpisodeResolver -> S3EpisodeResolver
revert and a stray _LEGACY_EMBODIMENT_ALIASES removal.
…configs
Ports the D=40cm / M=100 arc-tok model configs and required class code
onto the bf/7-configs config layout so the in-flight training jobs can
launch from this branch:
- Restore hpt_cotrain_enc_dec_base.yaml with _target_ paths pointing at
bf's module layout (algo.hpt.algo.HPT, models.stems.hpt_stems.*,
models.heads.hpt_heads.MultiBlockTransformerDecoder,
models.diffusion.denoising_nets.CrossTransformer).
- Point hpt_cotrain_mecka_flow_shared_head_arc.yaml back at
hpt_cotrain_enc_dec_base and update its _target_ paths.
- New: hpt_cotrain_mecka_flow_shared_head_arc_D40_M100{,_veldec,_velreadout}.yaml.
- Add MLPVelocityDecoder to egomimic/models/heads/hpt_heads.py and
FMPolicyWithVelDecoder / FMPolicyWithVelReadout to
egomimic/models/heads/fm_policy.py.
- Fix eval_arctok.yaml viz path (cartesian → cartesian/base).
- Add ``arc_tokenizer`` config group defaults to train_zarr_cartesian.yaml
so ``arc_tokenizer.min_distance_unit=…`` CLI overrides resolve.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- pushshapes arc-length tokenizer (egomimic/rldb/zarr/pushshapes_arc_tokenizer.py)
and pushshapes.get_keymap_hpt_arc for reading (M+1, 2) arc-tok windows.
- HPT closed-loop inference: expand_arc_chunk_to_time so the (M+1, D) chunk is
played back as a time-uniform buffer against the env; replan_at tracks the
variable expanded-buffer length.
- hydra config groups for the pusht arc-tok stack:
data/pushshapes/pusht/{circle,circle_arc}.yaml, evaluator/hpt/{pusht,pusht_arc}.yaml,
model/pusht/*, model/pusht_arc/*.
- Restored hpt_cotrain_mecka_flow_shared_head.yaml (non-arc baseline) with
bf/7-configs _target_ paths so mecka baselines resume from aniketh/arc.
- logger/wandb/base.yaml default project: zarr_test -> arc so arc-tok runs land
in rl2-group/arc automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adapts robot/rollout.py for arc-tokenized policies. Subclasses PolicyRollout to reuse the loader / obs transform / safety pipeline; overrides only the predict step to detokenize the model's (M+1, 8) arc-tok output into a (H, 14) time-uniform chunk before handing it to the controller.
Rewrites TokenizeBimanualArcLengthCartesian to produce (M+1, 14) with full xyz + ypr + grip per arm (was (M+1, 8) dropping rotation). Rotation is unconditionally supervised — no opt-out. Gripper padding for human aria data preserved as an existing transform option. - arc_length_tokenizer.py: ARC_TOK_PER_ARM_DIM=7, ARC_TOK_BIMANUAL_DIM=14; SLERP for waypoint ypr resample; vel row extended to 14 dims with per-axis mean angular velocity in ypr slots. - Model configs (arc, D20_M15, D40_M25, D40_M100, M15/M25/M50/M100 + veldec + velreadout): act_dim 8 -> 14, infer_ac_dims 8 -> 14, veldec output_dim 8 -> 14. - FMPolicyWithVelDecoder / FMPolicyWithVelReadout: act_dim default 14. - eval_arctok.py, rollout-arc.py, visualization/arc_tok_viz.py: shape asserts 8 -> 14; removed zero-fill of ypr; detokenize returns 14-dim including model-predicted rotation. - Data configs (arc_sweep_*, arc_tests_cotrain_arctok*, folding_clothes _arctok, folding_eva_fold_cotrain_arctok, eva_only_fold_arctok, aria_train_mecka_val_arctok): comment updates. - Embodiment keymap/transforms: docstring updates only; routing intact.
Ports the 9-run rotation-fixed setup into a first-class doc: - What each of the 9 runs is (arc-tok, veldec, velreadout, eva_only_arctok, three non-arc baselines). - Exact common overrides + arc-tok specifics + baseline specifics. - Description strings for wandb id resolution. - Launcher script paths and how to swap partitions. - Why all 9 are fresh (8-dim ckpt vs 14-dim head shape mismatch). - Known operational issues: billing quota kills, norm-stats cold start, partition availability. - Pre-fire verification checklist.
The action-transform refactor on transform_fixes replaced the single `mode:` key with action_mode / coord_frame / rotation_mode on Eva and Human. Arc's data configs were written before that landed, so once arc is stacked on abc they were passing an unknown `mode` kwarg. 121 transform_list blocks across 37 configs, mechanically: cartesian -> cartesian / camframe / euler cartesian_padded -> cartesian_gripper_padded / camframe / euler cartesian_wristframe_ypr -> cartesian / eef_frame / euler keypoints_*frame_ypr -> keypoints / camframe|eef_frame / euler arc_tokenizer_cartesian -> arc_tokenizer_cartesian / camframe / euler pushshapes keeps its own module-level `mode` vocabulary (arc_tokenizer, pad_only) — it was never part of the refactor and is left alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnBSsxfUNUZQQ9wBwxbkaf
abc gave Yam the refactored action_mode / coord_frame / rotation_mode signature but only the plain cartesian layout. arc's abc_fstshirt_mecka_freefold_arc_cotrain_D40_M100 config feeds yam_bimanual through the arc tokenizer, so it was passing min_distance_unit / resampled_vector_length into a method that had no such parameters. Routes through the same _append_arc_tokenizer helper Eva and Human use. Yam is already 14D with a real gripper, so unlike Human it needs no padding step before tokenizing. Also ports scripts/pixel_check.py, the last caller still on the old single-`mode` API. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnBSsxfUNUZQQ9wBwxbkaf
Human's arc mode was silently doing two things at once: padding the zero
gripper AND tokenizing. That made `arc_tokenizer_cartesian` mean something
different on Human than on Eva/Yam, and it did not parallel the existing
cartesian / cartesian_gripper_padded pair.
Splits them:
arc_tokenizer_cartesian tokenize the native layout
(Eva, Yam - real gripper, 14D)
arc_tokenizer_cartesian_gripper_padded pad, then tokenize
(Human - no gripper, 12D -> 14D)
Human's bare `arc_tokenizer_cartesian` now raises and points at the padded
variant instead of quietly padding. The 48 Human blocks across 24 data
configs plus scripts/pixel_check.py move to the explicit name, so no
pipeline changes shape - it is the same transform list, honestly named.
Eva's 44 blocks are untouched.
Also guards rotation_mode at build time. The tokenizer's chunk layout is a
hard-coded 14D [xyz(3), ypr(3), grip(1)] x 2 and it SLERPs the ypr slots,
so quat (16D) and 6D (20D) were only ever going to fail - previously on
the first batch, deep inside a run, now when the transform list is built.
Verified: (M+1, 14) out, arc length 0.1998m for D=0.20, gripper slots
zero, rotation supervised; all 262 transform_lists across 101 data configs
still instantiate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnBSsxfUNUZQQ9wBwxbkaf
One evaluator serves arc-tokenized and time-indexed runs. Arc detection is by row count (M waypoints plus a velocity token vs rollout_horizon steps), so a baseline chunk passes through instead of hitting the detokenizer. Every metric is arc-matched: clip to the first D metres of travel, resample to N points spaced uniformly in arc length, then score. Names carry no hint of the action space, so an arc run and its baseline twin plot on one axis. Scoring is EEF-frame and every metric is frame invariant. Position error survives a rigid transform, and rotation is reported as a geodesic angle, |log(R_pred R_gt^T)|, rather than a per-axis ypr difference: ypr is frame dependent and breaks at wraparound and gimbal lock, so it cannot be compared across runs. arc_matched_resample can now carry rotation through instead of resampling it and discarding it. Adds the D40/M100 experiment set: cotrain and ABC-only BC, arc and baseline, plus a 300M variant, all on the same evaluator.
AnikethCheluva
changed the base branch from
bf/7-configs
to
graphite-base/573
September 3, 2026 18:31
AnikethCheluva
force-pushed
the
aniketh/arc
branch
from
September 3, 2026 18:31
6617e22 to
e3b351d
Compare
AnikethCheluva
force-pushed
the
graphite-base/573
branch
from
September 3, 2026 18:31
a4d1839 to
19b84c1
Compare
AnikethCheluva
changed the base branch from
graphite-base/573
to
aniketh/abc
September 3, 2026 18:31
Collaborator
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.