Preserve storage metadata and limit Python support to 3.12-3.14 - #40
Merged
Conversation
`get_metadata` reported `last_updated: None` for a key that had just been
updated. `normalize_storage_metadata` merged the provider's `custom` block over
the storage record with `{**metadata, **custom}`, and `HiveStorage.write` stamps
top-level `last_updated` itself while filing the caller's dict under `custom`.
The initial store hands it a populated block; the incremental `update()` path
hands it one whose `last_updated`, `start_date` and `end_date` are None. The
blanket merge let those nulls win over the record's own correct values.
`custom` still overrides the record - that is what it is for - but only where it
has a value. A key that exists nowhere else keeps its null rather than being
dropped, so a caller that reads it still gets None instead of a KeyError.
Found from 02_financial_data_universe/18_data_management, which prints the
timestamp under "Verify data is current" and was showing None for all three
symbols. Four tests; the two that pin the defect fail without this change.
Suite: 3593 passed, 279 deselected.
stefan-jansen
changed the title
added 6 commits
August 10, 2026 11:21
stefan-jansen
added a commit
to stefan-jansen/machine-learning-for-trading
that referenced
this pull request
Aug 10, 2026
18_data_management prints the stored timestamp under "Verify data is current" and showed `last updated: None` for all three symbols. The value was on disk; normalize_storage_metadata merged the provider's `custom` block over the storage record wholesale, and the incremental update() path writes a block whose `last_updated` is null, so the null won over the record's own stamp. Fixed in ml4t-data 0.1.2 (ml4t/data#40, merge d558dc6): `custom` still overrides the record, but only where it has a value. 0.1.2 also declares Requires-Python <3.15,>=3.12. AAPL None -> 2026-08-10 16:49:30.467769+00:00 MSFT None -> 2026-08-10 16:49:31.170910+00:00 GOOGL None -> 2026-08-10 16:49:31.785906+00:00 Re-executed and re-stamped: no errors, no unexecuted cells, no deprecations.
stefan-jansen
added a commit
to stefan-jansen/machine-learning-for-trading
that referenced
this pull request
Aug 10, 2026
* Take ml4t-data to 0.1.1 and un-break the two chapter 02 notebooks #510 moved every ml4t-* library to its stable release and left two teaching notebooks failing against the new pins, recorded in that commit's message. Both needed library changes rather than notebook changes, which is what ml4t-data 0.1.1 (cc20d221, tagged v0.1.1) carries: - 13_data_quality_framework: the anomaly detectors rejected a pl.Date timestamp column, so every detector cell raised. 0.1.1 accepts it. - 18_data_management: HiveStorage encodes the partition key for filesystem safety and commits each write into a new generation directory, so the layout is no longer readable by globbing the store root - the notebook's rglob("*.parquet") walk found the files but could not recover the symbol or the period from the path. 0.1.1 adds HiveStorage.partitions(), which reports label and size_bytes per partition, and the two cells that inspected the directory tree now ask the store instead. Both notebooks execute clean and are re-stamped. AAPL reports 24 partitions over the two-year load, one per calendar month, which is what the prose below the cell already said the layout would be. * Use the negative-price policy instead of the boolean it replaced 13_data_quality_framework asked OHLCVValidator for check_negative_prices=True, which ml4t-data migrates to negative_price_policy="forbid" and warns about. The warning was in the executed page: a notebook that teaches the validation API was showing the reader a DeprecationWarning naming a different one. Same behaviour - rules.py maps True to "forbid" - so no validation result moves. * Take ml4t-data to 0.1.2, so the freshness check reports a time 18_data_management prints the stored timestamp under "Verify data is current" and showed `last updated: None` for all three symbols. The value was on disk; normalize_storage_metadata merged the provider's `custom` block over the storage record wholesale, and the incremental update() path writes a block whose `last_updated` is null, so the null won over the record's own stamp. Fixed in ml4t-data 0.1.2 (ml4t/data#40, merge d558dc6): `custom` still overrides the record, but only where it has a value. 0.1.2 also declares Requires-Python <3.15,>=3.12. AAPL None -> 2026-08-10 16:49:30.467769+00:00 MSFT None -> 2026-08-10 16:49:31.170910+00:00 GOOGL None -> 2026-08-10 16:49:31.785906+00:00 Re-executed and re-stamped: no errors, no unexecuted cells, no deprecations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
None.>=3.12,<3.15as the distribution contract.>=2.12,<3while disallowing prereleases in the locked development and release environment.Why
get_metadatareportedlast_updated: Noneimmediately after an incremental update even though storage had committed the correct timestamp. The read path let null provider metadata replace committed values. The incremental writer also rebuilt metadata without the mergedstart_dateandend_date, and replaced the stored provider, exchange, and calendar with defaults.The 0.1 releases allowed installation on Python 3.15 through
Requires-Python >=3.12, while the project advertised only 3.12 through 3.14. Python 3.15 jobs installed and imported successfully but failed broadly inside Polars on all three operating systems. Pydantic support for 3.15 is prerelease-only. The installer metadata and required CI matrix now match the stable support statement.User impact
Freshness metadata reports the committed timestamp and complete merged range after updates. Provider identity and exchange/calendar assignments remain stable when an update does not override them. Release 0.1.2 rejects Python 3.15; the README notes that unpinned 3.15 installations may still select 0.1.0 or 0.1.1 because their published metadata cannot be changed.
Verification
origin/mainand passed on this branch.start_date, andend_datebefore the writer fix.ty check: passed.Requires-Python: <3.15,>=3.12;pydantic<3,>=2.12.