docs(0053): close 0053 code-complete + spawn 0088 backfill run tracker

lore/1-tasks/active/0088_FEATURE_soroban-backfill-run-tracker/README.md

@@ -0,0 +1,107 @@
+---
+id: "0088"
+title: "Execute + track the historical backfill run (SDEX + Soroban AMM) to completion"
+type: FEATURE
+status: active
+related_adr: ["0005", "0009"]
+related_tasks: ["0053", "0028", "0082"]
+tags: [layer-infra, priority-medium, effort-large, milestone-M1, backfill, soroban, sdex, operational, tracker]
+milestone: 1
+links:
+  - "../../../docs/runbooks/continue-soroban-backfill.md"
+  - "../../../docs/runbooks/running-ingestion-components.md"
+history:
+  - date: 2026-07-08
+    status: active
+    who: okarcz
+    note: >
+      Spawned from 0053 operational tail. 0053's code + tests + runbook are
+      complete and merged (PR #72); the only remainder is executing the
+      multi-day two-range archive run and the post-run data spot-check, which
+      this task owns and tracks. Backfill currently PAUSED by operator decision,
+      to be resumed (ideally from a us-east-2 EC2). First clean forward chunk
+      [50688000, 51007999] already landed 2026-07-07.
+---
+
+# Execute + track the historical backfill run (SDEX + Soroban AMM)
+
+## Summary
+
+Operational tracker for running the combined single-pass historical backfill
+(SDEX + Soroban AMM) to completion and confirming the data. All code, tests,
+and the runbook landed under **task 0053** (merged, PR #72) — **no code work
+remains**. This task exists solely to execute the multi-day archive run,
+resume/track its chunked progress, and close the two operator-run data ACs that
+0053 could not confirm without the real run.
+
+**How to run it:** follow **[`docs/runbooks/continue-soroban-backfill.md`](../../../docs/runbooks/continue-soroban-backfill.md)**
+(first-timer resume guide — resumes from `soroban_amm.current_ledger` in 320k
+chunks, each direct-written to Hetzner and run to completion). Component/run
+context: [`docs/runbooks/running-ingestion-components.md`](../../../docs/runbooks/running-ingestion-components.md) §1.
+
+## Context
+
+The run is **download-bound** (~184 KB/ledger, ~4.8 MB/s, ~64 min/100k at home)
+and large: the combined remainder is ~2.3 TB / ~5.6 days continuous, and the
+pre-Soroban SDEX tail is ~4× larger again. The runbook strongly recommends a
+**us-east-2 EC2** (same region as the `aws-public-blockchain` bucket) to
+collapse download time before committing weeks of home bandwidth. The operator
+paused the run by choice; it will be resumed later.
+
+**Run order (from 0053):** combined `[activation, 63352611]` first, then
+sdex-only `[1, activation−1]` with `--tip <live tip>`.
+
+**`--end` floor = 63352611** (SDEX live-ingestion floor − 1) to avoid same-source
+minute overlap with live in `ReplacingMergeTree` (which silently undercounts, not
+duplicates — see [[backfill-live-no-code-coordination]]). Backfill now
+contaminates `min(sdex ledger)` in `price_ohlcv_1m`, so **re-derive the live
+floor from a non-backfilled source (or the live cursor) right before the final
+combined chunk**; it only moves forward, so 63352611 stays a safe ceiling
+meanwhile.
+
+## Progress log
+
+| Range | Status |
+|-------|--------|
+| `[activation, 50687999]` | data + `pool_registry` landed; run fatal-exited on the 0087 router guard (fixed + merged, PR #92). |
+| `[50688000, 51007999]` | ✅ first clean forward chunk (2026-07-07) — 320k ledgers, SDEX 17.22M candles, AMM=0/oracle=0 (expected early epoch), 0 new unresolved. ~3.4 h, 59 GB. |
+| `[51008000, 63352611]` combined remainder | ⏳ ~12.3M ledgers ≈ ~2.3 TB / ~5.6 days continuous. |
+| `[1, 50457423]` pre-Soroban SDEX tail | ⏳ ~50.5M ledgers (~4× larger); run `--mode sdex-only` after combined, `--tip <live tip>`. |
+
+> Update this table as chunks land (ledger range, candle counts, duration, any
+> new unresolved pools). `GET /backfill/status` reports truthful monotonic
+> progress for both streams.
+
+## Acceptance Criteria
+
+- [ ] Combined range `[activation, 63352611]` fully written to Hetzner over the
+      chunked forward passes; the unregistered-pool guard never fires on a clean
+      chunk; `pool_registry` persisted at each run-end.
+- [ ] Pre-Soroban SDEX tail `[1, activation−1]` completed with `--mode sdex-only`
+      (`--tip <live tip>`); union coverage `[1, tip]`, no ledger downloaded twice,
+      no same-minute overlap with live (final-chunk floor re-derived before launch).
+- [ ] `GET /backfill/status` reports truthful, monotonic progress throughout;
+      `sdex_archive`→`completed` and `soroban_amm`→`completed`, `status='paused'`
+      set at combined-run completion.
+- [ ] **OHLCV for Soroswap pairs verifiable for Nov 2023 dates** (Tranche-1 AC) —
+      data spot-check after the run lands. *(Inherited from 0053 — its last open AC.)*
+- [ ] Docker-gated CH integration tests greened once locally against prod-pinned
+      ClickHouse (`candles_it`, `pool_registry_it`, `progress_it`). *(0053 left
+      these to run once against a local CH.)*
+
+## Out of scope
+
+- Any code changes to the backfill engine, sink, or progress writer — all landed
+  in 0053 (PR #72) and dependents (#92). If a bug surfaces during the run, spawn
+  a fix task; this tracker stays operational.
+- Live-ingestion / periodic-worker verification — that is task **0082**.
+
+## Notes
+
+- Standing rules keep this operator-run against real infra (archive fetch +
+  Hetzner direct-write); not executed autonomously
+  ([[feedback-prepare-not-deploy]], [[feedback-local-only-no-prod-data]] —
+  note the read-only public-ledger fetch exception).
+- Measured throughput + EC2 rationale: [[local-backfill-throughput-measured]].
+- Prod CH access for progress checks (read-only SQL via `docker exec`):
+  [[hetzner-ch-prod-ssh-access]].

lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md → lore/1-tasks/archive/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md

@@ -2,9 +2,9 @@
 id: "0053"
 title: "Combined single-pass historical backfill (SDEX + Soroban AMM) — full-chain, forward-discovery, dual-stream progress"
 type: FEATURE
-status: active
+status: completed
 related_adr: ["0001", "0003", "0004", "0007", "0009"]
-related_tasks: ["0017", "0028", "0034", "0037", "0048", "0052", "0051", "0058", "0026", "0060", "0069", "0073", "0063"]
+related_tasks: ["0017", "0028", "0034", "0037", "0048", "0052", "0051", "0058", "0026", "0060", "0069", "0073", "0063", "0088"]
 tags: [layer-indexing, priority-high, effort-large, milestone-M1, stream-1, single-pass, rust, cli, workstation, clickhouse, soroban, amm, soroswap, aquarius, phoenix, sdex]
 milestone: 1
 links:
@@ -143,6 +143,20 @@ history:
       estimate). Also greened the 3 Docker-gated CH ITs locally
       (candles/pool_registry/progress). Resume workflow validated: `--start =
       soroban_amm.current_ledger + 1`, +320k steps, `--end` floor 63352611.
+  - date: 2026-07-08
+    status: completed
+    who: okarcz
+    note: >
+      DONE (code-complete). All implementation, tests, code-review fixes, and the
+      runbook landed and merged (PR #72; 0087 guard fix #92). The only remainder
+      was the multi-day operational archive run + the Nov-2023 Soroswap OHLCV
+      data spot-check — no code work — which is intentionally paused by the
+      operator (to be resumed, ideally from a us-east-2 EC2). That operational
+      tail is **moved to new task 0088** (backfill run tracker), which points to
+      docs/runbooks/continue-soroban-backfill.md. Closing 0053 as code-complete;
+      0088 tracks the run to completion. The Docker-gated CH ITs were greened
+      locally on 2026-07-07 (see prior entry); the one open AC (Soroswap Nov-2023
+      OHLCV) carries to 0088.
 ---
 
 # Combined single-pass historical backfill (SDEX + Soroban AMM)
@@ -378,8 +392,11 @@ and teardown.
       streams (no SDEX under-report). *Monotonic direction-aware `current_ledger`
       + non-downgrade of `completed` (code-review findings 1–4) + strictly-
       increasing RMT version; read-side `progress_pct` exposure is task 0073.*
-- [ ] **(operational)** OHLCV for Soroswap pairs verifiable for Nov 2023 dates
-      (Tranche 1 AC) — requires the real archive run + a data spot-check.
+- [→] **(operational, moved to 0088)** OHLCV for Soroswap pairs verifiable for
+      Nov 2023 dates (Tranche 1 AC) — requires the real archive run + a data
+      spot-check. *Code + tests + runbook complete here; the multi-day run and
+      this data check are tracked by task **0088** (backfill run tracker), so
+      0053 closes as code-complete.*
 - [x] Idempotent re-run of `push` (no duplicate rows after merge). (CH ITs:
       `candles_it` per-source + re-write, `progress_it`, `pool_registry_it` —
       all assert stable `count() … FINAL`.)
@@ -402,7 +419,10 @@ final code gaps — **`paused` status between runs** and the **activation
 minute-alignment guard** — are implemented. 54 unit tests pass; `cargo check
 --workspace`/`--features aws-mtls`/`fmt`/clippy clean.
 
-**Remaining — operational only (keeps this task active):**
+**Remaining — operational only (MOVED to task 0088 on 2026-07-08; 0053 closed
+code-complete).** The multi-day run + data check no longer gate this task — see
+`0088_FEATURE_soroban-backfill-run-tracker` and
+`docs/runbooks/continue-soroban-backfill.md`:
 
 1. **Execute the two-range run** against the pubnet archive and direct-write to
    Hetzner: combined `[activation, tip]` then sdex-only `[1, activation-1]`