refactor(config): organize all Hydra configs by family and task - #551
Draft
ElmoPA wants to merge 12 commits into
Draft
refactor(config): organize all Hydra configs by family and task#551ElmoPA wants to merge 12 commits into
ElmoPA wants to merge 12 commits into
Conversation
elmo/batchflow-core forked from main on 2026-05-18 and never restacked: 170 commits ahead, 73 behind, carrying code main had since deleted. Rebasing it would replay 170 commits to move files that mostly cannot conflict, so this ports the net result onto current main instead. 88% of the change is purely additive -- whole new subsystems (models/hnet, models/diffusion, eval/core, the pipeline, 223 model configs) with no counterpart on main to fight with. Composition: 775 added, 25 renamed (batchflow's file reorganisation), 13 deleted, 14 modified. WHERE THE LINEAGES DISAGREED, MAIN WINS. Almost every conflict traces to one main commit, 0bde7a7 "mandatory zarr.json intrinsics + collapse human embodiments", which deleted CameraTransforms and renumbered the embodiments. batchflow's changes to those files exist only to serve an API that is gone, so they are dropped rather than reconciled: * embodiment.py takes main's collapsed enum (HUMAN_* 1-3, EVA_* 4-6) and re-adds PUSHSHAPES_SIM 15 / _STICK 16 / _SMALL_CIRCLE 17. Those IDs are pinned because trained checkpoints and collected datasets encode them -- renumbering would silently re-route every existing pushshapes result. * data_schematic/default.yaml and hpt.yaml collapse aria_/mecka_/scale_bimanual into one human_bimanual block. Verified non-destructive: aria was a strict superset -- it alone carried state_keypoints and actions_keypoints -- and nothing was unique to mecka or scale. * the CameraTransforms-era hpt/act/pi0.5 configs, viz_utils, rollout and egomimicUtils keep main's per-episode-intrinsics versions. Three files needed a union rather than a choice, done as a 3-way merge against the fork point so disjoint additions from both sides survive: * zarr_dataset_multi.py keeps main's SafeS3EpisodeResolver, EvenStrideDataset, _evenly_spaced_indices, _jpeg_probe_failed and the intrinsics property, plus batchflow's _read_span, _annotations_for_span and LocalEpisodeResolverWithEmbodimentOverride. Ten ported files depend on that resolver and zarr_dataset_packed is written around _read_span. Both genuine overlaps resolved to main. ZarrEpisode._get_store is kept: the merged __init__ carries its _pid state and read() calls it, and zarr v3 uses asyncio internally so the store is not fork-safe under DataLoader workers. * action_chunk_transforms.py takes batchflow's DeltaAction alongside main's PadGripperZeros. * trainHydra.py takes wiring from both; no overlap. Two of git's 27 detected renames were SKIPPED as spurious -- it matched empty __init__.py files across unrelated trees, e.g. egomimic/rldb/scripts/__init__.py to Tsimulation/__init__.py, and applying them would have deleted live packages. Likewise 33 of the 46 net "deletions" are files main added after the fork rather than batchflow deletions; only the 13 batchflow actually removed are applied. Verified by importing every module in the tree on a compute node: 209 import clean. The 8 that do not are pre-existing or environmental, none introduced here: missing optional deps (streamlit, oculus_reader, ppadb, robot_utils), a hardcoded PACE path, a test importing build_human_bimanual_transform_list which exists on neither branch since transform lists moved onto the embodiment classes, and three modules that each register a global OmegaConf "eval" resolver without replace=True. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012V58H37tmcvgDthELMd5Xk
This was referenced Aug 7, 2026
Contributor
Author
…olves
The reorganised trainer configs reference their parents by absolute path
(`- /trainer/base/default`), which merges the referenced config at package
`trainer.base.default` rather than into the current node. Composing
`trainer=distributed/ddp` then yields a trainer with no `_target_`, no
max_epochs/min_epochs, no precision and no limit_train_batches, so it cannot
be instantiated at all.
Verified by composing train_zarr_cartesian both ways:
with @_here_: trainer.max_epochs=2000, _target_ present
without @_here_: trainer.max_epochs=ABSENT, _target_ absent, plus an
orphan nested trainer.trainer node
ElmoPA
changed the base branch from
sim/converge-layout-rebase
to
graphite-base/551
August 12, 2026 01:05
ElmoPA
force-pushed
the
graphite-base/551
branch
from
August 12, 2026 01:05
1910edd to
54142d0
Compare
This was referenced Aug 12, 2026
ElmoPA
force-pushed
the
graphite-base/551
branch
from
August 21, 2026 03:19
21647be to
1910edd
Compare
ElmoPA
changed the base branch from
graphite-base/551
to
sim/converge-layout-rebase
August 21, 2026 03:19
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
refactor(config): organize all Hydra configs by family and task
chore(config): remove generated egg metadata
fix(config): correct organized dataset example paths
refactor(config): organize shared Hydra groups