diff --git a/lore/1-tasks/active/0088_FEATURE_soroban-backfill-run-tracker/README.md b/lore/1-tasks/active/0088_FEATURE_soroban-backfill-run-tracker/README.md new file mode 100644 index 0000000..c586014 --- /dev/null +++ b/lore/1-tasks/active/0088_FEATURE_soroban-backfill-run-tracker/README.md @@ -0,0 +1,107 @@ +--- +id: "0088" +title: "Execute + track the historical backfill run (SDEX + Soroban AMM) to completion" +type: FEATURE +status: active +related_adr: ["0005", "0009"] +related_tasks: ["0053", "0028", "0082"] +tags: [layer-infra, priority-medium, effort-large, milestone-M1, backfill, soroban, sdex, operational, tracker] +milestone: 1 +links: + - "../../../docs/runbooks/continue-soroban-backfill.md" + - "../../../docs/runbooks/running-ingestion-components.md" +history: + - date: 2026-07-08 + status: active + who: okarcz + note: > + Spawned from 0053 operational tail. 0053's code + tests + runbook are + complete and merged (PR #72); the only remainder is executing the + multi-day two-range archive run and the post-run data spot-check, which + this task owns and tracks. Backfill currently PAUSED by operator decision, + to be resumed (ideally from a us-east-2 EC2). First clean forward chunk + [50688000, 51007999] already landed 2026-07-07. +--- + +# Execute + track the historical backfill run (SDEX + Soroban AMM) + +## Summary + +Operational tracker for running the combined single-pass historical backfill +(SDEX + Soroban AMM) to completion and confirming the data. All code, tests, +and the runbook landed under **task 0053** (merged, PR #72) — **no code work +remains**. This task exists solely to execute the multi-day archive run, +resume/track its chunked progress, and close the two operator-run data ACs that +0053 could not confirm without the real run. + +**How to run it:** follow **[`docs/runbooks/continue-soroban-backfill.md`](../../../docs/runbooks/continue-soroban-backfill.md)** +(first-timer resume guide — resumes from `soroban_amm.current_ledger` in 320k +chunks, each direct-written to Hetzner and run to completion). Component/run +context: [`docs/runbooks/running-ingestion-components.md`](../../../docs/runbooks/running-ingestion-components.md) §1. + +## Context + +The run is **download-bound** (~184 KB/ledger, ~4.8 MB/s, ~64 min/100k at home) +and large: the combined remainder is ~2.3 TB / ~5.6 days continuous, and the +pre-Soroban SDEX tail is ~4× larger again. The runbook strongly recommends a +**us-east-2 EC2** (same region as the `aws-public-blockchain` bucket) to +collapse download time before committing weeks of home bandwidth. The operator +paused the run by choice; it will be resumed later. + +**Run order (from 0053):** combined `[activation, 63352611]` first, then +sdex-only `[1, activation−1]` with `--tip `. + +**`--end` floor = 63352611** (SDEX live-ingestion floor − 1) to avoid same-source +minute overlap with live in `ReplacingMergeTree` (which silently undercounts, not +duplicates — see [[backfill-live-no-code-coordination]]). Backfill now +contaminates `min(sdex ledger)` in `price_ohlcv_1m`, so **re-derive the live +floor from a non-backfilled source (or the live cursor) right before the final +combined chunk**; it only moves forward, so 63352611 stays a safe ceiling +meanwhile. + +## Progress log + +| Range | Status | +|-------|--------| +| `[activation, 50687999]` | data + `pool_registry` landed; run fatal-exited on the 0087 router guard (fixed + merged, PR #92). | +| `[50688000, 51007999]` | ✅ first clean forward chunk (2026-07-07) — 320k ledgers, SDEX 17.22M candles, AMM=0/oracle=0 (expected early epoch), 0 new unresolved. ~3.4 h, 59 GB. | +| `[51008000, 63352611]` combined remainder | ⏳ ~12.3M ledgers ≈ ~2.3 TB / ~5.6 days continuous. | +| `[1, 50457423]` pre-Soroban SDEX tail | ⏳ ~50.5M ledgers (~4× larger); run `--mode sdex-only` after combined, `--tip `. | + +> Update this table as chunks land (ledger range, candle counts, duration, any +> new unresolved pools). `GET /backfill/status` reports truthful monotonic +> progress for both streams. + +## Acceptance Criteria + +- [ ] Combined range `[activation, 63352611]` fully written to Hetzner over the + chunked forward passes; the unregistered-pool guard never fires on a clean + chunk; `pool_registry` persisted at each run-end. +- [ ] Pre-Soroban SDEX tail `[1, activation−1]` completed with `--mode sdex-only` + (`--tip `); union coverage `[1, tip]`, no ledger downloaded twice, + no same-minute overlap with live (final-chunk floor re-derived before launch). +- [ ] `GET /backfill/status` reports truthful, monotonic progress throughout; + `sdex_archive`→`completed` and `soroban_amm`→`completed`, `status='paused'` + set at combined-run completion. +- [ ] **OHLCV for Soroswap pairs verifiable for Nov 2023 dates** (Tranche-1 AC) — + data spot-check after the run lands. *(Inherited from 0053 — its last open AC.)* +- [ ] Docker-gated CH integration tests greened once locally against prod-pinned + ClickHouse (`candles_it`, `pool_registry_it`, `progress_it`). *(0053 left + these to run once against a local CH.)* + +## Out of scope + +- Any code changes to the backfill engine, sink, or progress writer — all landed + in 0053 (PR #72) and dependents (#92). If a bug surfaces during the run, spawn + a fix task; this tracker stays operational. +- Live-ingestion / periodic-worker verification — that is task **0082**. + +## Notes + +- Standing rules keep this operator-run against real infra (archive fetch + + Hetzner direct-write); not executed autonomously + ([[feedback-prepare-not-deploy]], [[feedback-local-only-no-prod-data]] — + note the read-only public-ledger fetch exception). +- Measured throughput + EC2 rationale: [[local-backfill-throughput-measured]]. +- Prod CH access for progress checks (read-only SQL via `docker exec`): + [[hetzner-ch-prod-ssh-access]]. diff --git a/lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md b/lore/1-tasks/archive/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md similarity index 94% rename from lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md rename to lore/1-tasks/archive/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md index 14555aa..4431a1b 100644 --- a/lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md +++ b/lore/1-tasks/archive/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md @@ -2,9 +2,9 @@ id: "0053" title: "Combined single-pass historical backfill (SDEX + Soroban AMM) — full-chain, forward-discovery, dual-stream progress" type: FEATURE -status: active +status: completed related_adr: ["0001", "0003", "0004", "0007", "0009"] -related_tasks: ["0017", "0028", "0034", "0037", "0048", "0052", "0051", "0058", "0026", "0060", "0069", "0073", "0063"] +related_tasks: ["0017", "0028", "0034", "0037", "0048", "0052", "0051", "0058", "0026", "0060", "0069", "0073", "0063", "0088"] tags: [layer-indexing, priority-high, effort-large, milestone-M1, stream-1, single-pass, rust, cli, workstation, clickhouse, soroban, amm, soroswap, aquarius, phoenix, sdex] milestone: 1 links: @@ -143,6 +143,20 @@ history: estimate). Also greened the 3 Docker-gated CH ITs locally (candles/pool_registry/progress). Resume workflow validated: `--start = soroban_amm.current_ledger + 1`, +320k steps, `--end` floor 63352611. + - date: 2026-07-08 + status: completed + who: okarcz + note: > + DONE (code-complete). All implementation, tests, code-review fixes, and the + runbook landed and merged (PR #72; 0087 guard fix #92). The only remainder + was the multi-day operational archive run + the Nov-2023 Soroswap OHLCV + data spot-check — no code work — which is intentionally paused by the + operator (to be resumed, ideally from a us-east-2 EC2). That operational + tail is **moved to new task 0088** (backfill run tracker), which points to + docs/runbooks/continue-soroban-backfill.md. Closing 0053 as code-complete; + 0088 tracks the run to completion. The Docker-gated CH ITs were greened + locally on 2026-07-07 (see prior entry); the one open AC (Soroswap Nov-2023 + OHLCV) carries to 0088. --- # Combined single-pass historical backfill (SDEX + Soroban AMM) @@ -378,8 +392,11 @@ and teardown. streams (no SDEX under-report). *Monotonic direction-aware `current_ledger` + non-downgrade of `completed` (code-review findings 1–4) + strictly- increasing RMT version; read-side `progress_pct` exposure is task 0073.* -- [ ] **(operational)** OHLCV for Soroswap pairs verifiable for Nov 2023 dates - (Tranche 1 AC) — requires the real archive run + a data spot-check. +- [→] **(operational, moved to 0088)** OHLCV for Soroswap pairs verifiable for + Nov 2023 dates (Tranche 1 AC) — requires the real archive run + a data + spot-check. *Code + tests + runbook complete here; the multi-day run and + this data check are tracked by task **0088** (backfill run tracker), so + 0053 closes as code-complete.* - [x] Idempotent re-run of `push` (no duplicate rows after merge). (CH ITs: `candles_it` per-source + re-write, `progress_it`, `pool_registry_it` — all assert stable `count() … FINAL`.) @@ -402,7 +419,10 @@ final code gaps — **`paused` status between runs** and the **activation minute-alignment guard** — are implemented. 54 unit tests pass; `cargo check --workspace`/`--features aws-mtls`/`fmt`/clippy clean. -**Remaining — operational only (keeps this task active):** +**Remaining — operational only (MOVED to task 0088 on 2026-07-08; 0053 closed +code-complete).** The multi-day run + data check no longer gate this task — see +`0088_FEATURE_soroban-backfill-run-tracker` and +`docs/runbooks/continue-soroban-backfill.md`: 1. **Execute the two-range run** against the pubnet archive and direct-write to Hetzner: combined `[activation, tip]` then sdex-only `[1, activation-1]` diff --git a/lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/notes/G-100k-local-timing-and-schema-validation.md b/lore/1-tasks/archive/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/notes/G-100k-local-timing-and-schema-validation.md similarity index 100% rename from lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/notes/G-100k-local-timing-and-schema-validation.md rename to lore/1-tasks/archive/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/notes/G-100k-local-timing-and-schema-validation.md