fix(etfs): rebuild combined file from partitions, not stale aggregate - #46
Open
mikesdelaney-cmyk wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes ETFDataManager._regenerate_combined() so it actually rebuilds etf_universe.parquet from the individual ticker partitions.
Problem
_regenerate_combined() currently calls:
all_data = self.load_all()
but load_all() prefers the existing etf_universe.parquet when that file exists. This means _regenerate_combined() can read the stale combined file and write it straight back out, even after the individual ticker partitions have been updated successfully.
In practice, I reproduced this with the ML4T Chapter 25 ETF deployment loop:
individual ticker partitions advanced through 2026-08-20
ETFDataManager.update() reported new rows
_regenerate_combined() ran
the combined aggregate still ended at 2025-12-31
Deleting the combined file caused load_all() to fall back to load_symbols(...), after which the aggregate rebuilt correctly. That isolated the issue to the cache-preferring behavior of load_all().
Fix
Change _regenerate_combined() to load the configured ticker partitions directly:
all_data = self.load_symbols(self.config.get_all_symbols())
This matches the method docstring (“Regenerate combined file from individual ticker files”) and uses the same partition-loading path that load_all() already falls back to when the aggregate is absent.
Tests
Added regression coverage for:
rebuilding when an existing aggregate is stale
restoring symbols present in ticker partitions but missing from the aggregate
Targeted test run:
37 passed, 2 warnings
End-to-end verification
I installed this branch into the machine-learning-for-trading project via a SHA-pinned uv Git source and reran the Chapter 25 ETF deployment loop with the existing frozen etf_universe.parquet still present.
Before the fix, the combined file remained at:
470,662 rows, ending 2025-12-31
After the fix, the normal refresh path rebuilt it successfully without deleting the aggregate first:
ml4t-data update: 318 new rows across 2 symbols
Prices: 486,552 rows x 100 symbols, 2006-01-03 to 2026-08-20
No changes were made to the deployment logic itself; this PR only fixes aggregate regeneration inside ml4t-data.