docs(lore-0090): backfill history-loss — preroll + cleanup coordination#103
Open
karczuRF wants to merge 5 commits into
Open
docs(lore-0090): backfill history-loss — preroll + cleanup coordination#103karczuRF wants to merge 5 commits into
karczuRF wants to merge 5 commits into
Conversation
Backfill writes only price_ohlcv_1m, a 7-day-retention feeder; the live rollup MVs are refreshable/2h-windowed and ignore historical rows, so backfilled candles are partition-dropped un-rolled and the coarse forever-tables the BE consumer reads stay empty. Captures evidence and the corrected disable-cleanup -> backfill -> preroll -> re-enable flow. Blocks 0088.
Diagnostic example: decodes an archive ledger and runs the real extract_trades -> tick -> candle pipeline, proving the extractor produces candles for pre-floor ledgers (loss is downstream).
…backfill Self-contained, no-context-assumed procedure: stop backfill -> disable cleanup -> reset progress -> re-run into 1m -> preroll into permanent tables -> verify -> re-enable cleanup, with per-step confirm checks.
…nbook Disabling cleanup holds the full 1m history transiently (~5-9 GB measured); add a GO/NO-GO gate (Avail >= 20 GB) before disabling, with the per-chunk preroll fallback if the shared cluster is short on disk.
…ata) Move 0088 active -> blocked. Running it surfaced that backfilled candles land only in the 7-day price_ohlcv_1m and are dropped before reaching the permanent coarse tables (root cause + fix in 0090). Resume after 0090.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
While running the 0088 historical backfill, discovered that backfilled candle history is silently lost. This PR records the root cause + fix as lore task 0090 and adds a diagnostic tool. No production code is changed here — 0090 is a blocker for 0088.
Root cause (measured on ch-prod-01)
sdex-backfillwrites candles only toprices.price_ohlcv_1m— a 7-day-retention feeder, not a store of record.schema/rollups.sql) are refreshable, LIVE-only,now() - INTERVAL 2 HOURwindowed, so they deliberately ignore historical/backfilled rows. History must be rolled up byschema/preroll.sqlinstead.prices-{env}-cleanup,cron(0 2 * * *)) drops historical1mpartitions nightly.1mdata is written, never pre-rolled, and partition-dropped un-rolled. The coarse forever-tables (1h/4h/1d/1w/1M) the BE consumer reads are empty.The extractor is proven correct (archive ledger
51050000→ 133 trades → 114 candles through the real pipeline); the loss is entirely downstream at the destination.Contents
lore/1-tasks/backlog/0090_…md— root cause, evidence, and the correcteddisable-cleanup → backfill → preroll → verify → re-enableworkflow + acceptance criteria.packages/prices-ingest-core/examples/decode_probe.rs— diagnostic: decode an archive ledger and run the real extract→tick→candle pipeline.Follow-ups (in the task)
docs/runbooks/continue-soroban-backfill.mdwith the preroll + cleanup steps.1mrow ~1.3 days stale).