diff --git a/docs/runbooks/continue-soroban-backfill.md b/docs/runbooks/continue-soroban-backfill.md new file mode 100644 index 0000000..f75112f --- /dev/null +++ b/docs/runbooks/continue-soroban-backfill.md @@ -0,0 +1,324 @@ +# Runbook — Continue the Soroban historical backfill (chunk by chunk) + +**Audience:** anyone (incl. the BE team) running this for the first time. No prior +context assumed. Follow the steps top to bottom. + +## What this does + +We are backfilling historical price candles (`prices.*` on the Hetzner +ClickHouse) by replaying Stellar ledgers from the public archive. The whole job +is too big for one run, so we do it in **chunks**: each run indexes a slice of +ledgers, writes candles directly to Hetzner over mTLS, and records where it got +to. You just repeat: _find where the last run ended → run the next slice → +verify → repeat_, until the whole range is done. + +The tool is the `sdex-backfill` binary in this repo. In `combined` mode it +extracts **SDEX trades + Soroban AMM swaps + oracle prices** from a single +download of each ledger. + +### Where we are (update this line as you go) + +- **Done:** `[50457424, 51007999]` (the first tranche + first forward chunk). +- **Next combined chunk starts at:** `51008000`. +- **Remaining combined range:** up to `63352611` (the "floor" — see §7). +- **After that:** one `sdex-only` run over the pre-Soroban tail `[1, 50457423]`. + +> The authoritative "where did it end" is always the live number in ClickHouse +> (Step 3) — the line above is just a hint. + +--- + +## 0. Prerequisites + +- **A Linux machine to run on.** Strongly prefer an **AWS EC2 instance in + `us-east-2`** — see §1 (it makes the download ~20–50× faster). +- **Rust** (stable, via [rustup](https://rustup.rs)) — to build the binary. +- **AWS CLI v2** — used to download ledgers (`aws s3 sync`). +- **`jq`** — to split the mTLS bundle JSON into files. +- **Disk:** each 5-partition chunk downloads ~60 GB of ledgers, but the scratch + folder is cleaned per-partition as it goes, so **~20 GB free** is enough. (One + 64k-ledger partition is ~12 GB.) +- **Access to the prices AWS account** (`750702271865`) with permission to read + the writer secret in Step 2. If you don't have it, ask the prices-api owner to + grant `secretsmanager:GetSecretValue` on the secret named in Step 2, or to hand + you the three PEM files directly. +- **SSH access to the prod ClickHouse host** (to check progress) — key + `~/.ssh/sorban-prod_ed25519`, host `deploy@168.119.73.161`. + +--- + +## 1. Where to run it — pick the fastest region + +The public ledger archive lives in the S3 bucket **`aws-public-blockchain`, +which is in AWS region `us-east-2` (Ohio)**. Download bandwidth is the entire +bottleneck of this job (each ledger is ~180 KB and there are millions of them). + +- **Fastest by far:** run on an **EC2 instance in `us-east-2`**. Same-region S3 + reads are hundreds of MB/s and incur no cross-region data-transfer cost. A + chunk that takes ~3.4 h on a home connection can finish in minutes there. +- **Slow:** running from a laptop / office network pulls the data across the + internet at ~5 MB/s. Fine for one or two chunks; painful for the whole chain + (the remaining combined range alone is ~2.3 TB). + +Everything else (writing candles to Hetzner in Germany, fetching the secret from +`eu-central-1`) is tiny and location-independent — only the S3 download speed +matters, so **co-locate with the bucket in `us-east-2`.** + +If you launch an EC2 box: Amazon Linux or Ubuntu, an instance with good network +(e.g. `m6i.large`+), a ~40 GB disk, and install `rustup`, `awscli`, `jq`, `git`. + +--- + +## 2. Get AWS access and the mTLS write credential + +There are **two** AWS interactions, and they are different: + +1. **Downloading ledgers** — the archive bucket is public, so this needs **no + credentials at all**; the tool passes `--no-sign-request`. You do not sign in + for this. +2. **Fetching the write credential** — one-time, to get the certificate the + backfill uses to authenticate to the Hetzner ClickHouse. This needs AWS + credentials for the prices account. + +### 2a. Sign in to the prices AWS account (one-time, for the secret only) + +Use whatever your team uses for account `750702271865`. With AWS SSO that is: + +```bash +aws configure sso # first time only; set the start URL + region +aws sso login --profile soroban-explorer +``` + +(`soroban-explorer` is the conventional profile name — use yours if different.) + +### 2b. Fetch the writer mTLS bundle → three PEM files + +The backfill authenticates as the `prices_writer` ClickHouse user. Its +certificate/key/CA are stored as one JSON secret. Pull it and split it into +three files (the commands below **never print** the key to your terminal): + +```bash +mkdir -p ~/prices-mtls && cd ~/prices-mtls + +aws secretsmanager get-secret-value \ + --profile soroban-explorer \ + --region eu-central-1 \ + --secret-id prices/production/clickhouse-mtls-prices-ingestion-production \ + --query SecretString --output text > bundle.json + +jq -r .cert bundle.json > prices_writer.crt +jq -r .key bundle.json > prices_writer.key +jq -r .ca bundle.json > ca.crt + +chmod 600 prices_writer.key # lock down the private key +rm bundle.json # don't leave the combined secret lying around +``` + +> **Secret hygiene:** `prices_writer.key` is a private key. Never `cat`, echo, +> commit, or copy it anywhere. It only ever needs to sit in `~/prices-mtls` on +> the run box. Delete the folder when the whole backfill is finished. + +Keep these three paths handy — you'll point the tool at them in Step 5. + +--- + +## 3. Build the tool + +```bash +git clone stellar-prices-api # or: cd into your existing checkout +cd stellar-prices-api +git checkout develop && git pull --ff-only + +# The `aws-mtls` feature is REQUIRED to write to Hetzner. +cargo build --release -p sdex-backfill --features aws-mtls +``` + +The binary lands at `./target/release/sdex-backfill`. + +--- + +## 4. Find where the last run ended + +Ask the database for the forward watermark (read-only): + +```bash +ssh -i ~/.ssh/sorban-prod_ed25519 deploy@168.119.73.161 \ + "docker exec -i app-clickhouse-1 clickhouse-client --query='SELECT task_name, current_ledger, status FROM prices.backfill_progress FINAL FORMAT PrettyCompact'" +``` + +Look at the **`soroban_amm`** row. Its `current_ledger` is the last ledger done. +**Your next chunk starts at `current_ledger + 1`.** +(The `sdex_archive` row showing `paused` at `50457424` is expected — ignore it +until you reach the pre-Soroban tail in §8.) + +--- + +## 5. Run the next chunk + +**Pick the range.** Chunks are sized in whole 64,000-ledger partitions. Five +partitions = 320,000 ledgers ≈ 3–4 h on a home line, minutes on a us-east-2 EC2. + +- `START` = `soroban_amm.current_ledger + 1` (from Step 4). +- `END` = `START + 320000 - 1` (5 partitions), **but never past the floor + `63352611`** — see §7. +- `TIP` = the current live chain tip (so the progress % is meaningful): + +```bash +curl -s 'https://horizon.stellar.org/' \ + | python3 -c 'import sys,json;print(json.load(sys.stdin)["core_latest_ledger"])' +``` + +**Point at your cert files and run inside `tmux`** (so an SSH drop can't kill +the run — see §9 on why finishing matters): + +```bash +tmux new -s backfill # detach later with Ctrl-b then d; reattach: tmux attach -t backfill + +export CH_DOMAIN=ch.sorobanscan.rumblefish.dev +export MTLS_CERT_PATH=$HOME/prices-mtls/prices_writer.crt +export MTLS_KEY_PATH=$HOME/prices-mtls/prices_writer.key +export MTLS_CA_PATH=$HOME/prices-mtls/ca.crt + +./target/release/sdex-backfill \ + --mode combined \ + --start 51008000 \ + --end 51327999 \ + --tip \ + --transport hetzner \ + --verbose +``` + +(Replace `51008000 / 51327999` with your `START / END`, and `` with the +number from above.) A healthy start prints `pre-flight: all checks passed` and +`backfill starting … to_process: 5`. Then it downloads and indexes one partition +at a time, printing `partition indexing complete` with counts after each. When +the whole chunk is done you'll see `=== sdex-backfill complete ===`. + +> One **WARN** at startup — _"combined mode starts after activation…"_ — is +> **normal and expected** for every continuation chunk. It just means earlier +> pools are resolved from the saved registry, which is exactly what we want. + +--- + +## 6. Verify the chunk + +Run these (read-only) after `=== sdex-backfill complete ===`. Substitute your +chunk's `START`/`END`. + +```bash +# a) progress advanced to your END: +ssh -i ~/.ssh/sorban-prod_ed25519 deploy@168.119.73.161 \ + "docker exec -i app-clickhouse-1 clickhouse-client --query='SELECT task_name, current_ledger, status FROM prices.backfill_progress FINAL FORMAT PrettyCompact'" + +# b) all ledgers written (expect END-START+1, e.g. 320000; a few short = archive tail-lag, fine): +ssh -i ~/.ssh/sorban-prod_ed25519 deploy@168.119.73.161 \ + "docker exec -i app-clickhouse-1 clickhouse-client --query='SELECT count() FROM prices.backfill_sdex_ledgers WHERE sequence BETWEEN 51008000 AND 51327999'" + +# c) candles per source in the range: +ssh -i ~/.ssh/sorban-prod_ed25519 deploy@168.119.73.161 \ + "docker exec -i app-clickhouse-1 clickhouse-client --query='SELECT source, count() FROM prices.price_ohlcv_1m FINAL WHERE toUInt64(version) DIV 1000 BETWEEN 51008000 AND 51327999 GROUP BY source FORMAT PrettyCompact'" + +# d) any pools we could not resolve this chunk: +ssh -i ~/.ssh/sorban-prod_ed25519 deploy@168.119.73.161 \ + "docker exec -i app-clickhouse-1 clickhouse-client --query='SELECT * FROM prices.unresolved_pools FINAL FORMAT PrettyCompact'" +``` + +What "good" looks like: `soroban_amm.current_ledger` = your `END`; the ledger +count matches; `sdex` candles present. In the **early** Soroban era (below +~ledger 62,000,000) it is **normal** to see **no `aquarius`/`phoenix` rows and +zero oracle rows** — those AMMs/oracles weren't active yet; they appear in later +chunks. Rows in `unresolved_pools` with a `first_ledger` **below your chunk's +`START`** are old leftovers from earlier runs, not from this chunk. + +--- + +## 7. Repeat — and the "floor" you must not cross + +Go back to Step 4 and do the next chunk: new `START` = `END + 1`, new `END` = +`START + 320000 - 1`, fresh `TIP`. Keep stepping forward. + +**The floor.** The live ingestion service already writes candles from about +ledger **`63352612`** onward. The backfill must stop **below** that, or the two +would fight over the same minutes and undercount. So the **last** combined chunk +ends at exactly **`--end 63352611`** (do not go higher). All earlier chunks are +well below it, so you only think about the floor on the final chunk. + +> Because the backfill now writes early `sdex` candles too, you can no longer +> read the live floor with a simple `min()` over `sdex`. Before the **final** +> chunk only, re-confirm it from a source the backfill hasn't reached yet, e.g. +> `SELECT min(toUInt64(version) DIV 1000) FROM prices.price_ohlcv_1m FINAL WHERE source='aquarius'` — it never decreases, so `63352611` stays a safe ceiling meanwhile. + +Rough remaining size (home-line pace): the combined range to the floor is +~2.3 TB / ~5–6 days of continuous download; on a us-east-2 EC2 it's a small +fraction of that. + +--- + +## 8. Final step — the pre-Soroban tail + +Once a chunk reaches the floor and the combined range is done, run the classic +pre-Soroban ledgers **once** (SDEX only — there were no AMM pools before Soroban): + +```bash +TIP=$(curl -s 'https://horizon.stellar.org/' | python3 -c 'import sys,json;print(json.load(sys.stdin)["core_latest_ledger"])') + +./target/release/sdex-backfill \ + --mode sdex-only \ + --start 1 \ + --end 50457423 \ + --tip "$TIP" \ + --transport hetzner \ + --verbose +``` + +This is the largest range (~50M ledgers) — definitely one for a us-east-2 EC2. +It walks `sdex_archive` down to genesis; when it finishes, the backfill is +complete. + +--- + +## 9. If you need to stop and re-run + +**Stopping is safe.** You can `Ctrl-C` the run (or the machine can reboot, or the +network can drop) at any time without corrupting anything. Everything already +written stays valid: candles use a `ReplacingMergeTree`, so re-writing the same +ledger just replaces it — no duplicates, no double-counting. Completed ledgers +are recorded **per partition**, so the archive is never re-downloaded for work +already done. + +**To resume,** just do Step 4 again — read `soroban_amm.current_ledger`, set +`--start = current_ledger + 1`, and launch the next chunk as usual. (You can also +simply re-run the _same_ `--start/--end` you were on; already-finished ledgers +are skipped automatically. Either works.) + +**The one caveat — let each chunk finish.** The discovered pool registry is saved +to the database only at the _successful end_ of a chunk. If you kill a chunk +partway through, the AMM pools it discovered in that chunk aren't saved, and +because those ledgers are now marked done they won't be re-scanned on resume — so +a few pools created in that interrupted chunk can end up unregistered, and their +swaps get recorded in `prices.unresolved_pools` (visible, **not** silently lost). +In the early Soroban era this is a non-issue (no AMM activity yet). If it ever +happens in the AMM-active era and you want those pools back, the clean fix is to +run the pool-registry seeder once (`docs/runbooks/seed-pool-registry.md`) to +repopulate `prices.pool_registry`, then continue — no need to redo the whole +backfill. The simplest habit that avoids all of this: size chunks so each one +finishes in a sitting (5 partitions ≈ 3–4 h on a home line, far less on EC2) and +let it run to `=== sdex-backfill complete ===`. + +--- + +## Quick reference + +| Thing | Value | +| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- | +| Ledger bucket | `s3://aws-public-blockchain/v1.1/stellar/ledgers/pubnet/` — region **us-east-2**, anonymous (`--no-sign-request`) | +| Writer secret | `prices/production/clickhouse-mtls-prices-ingestion-production` (region `eu-central-1`, acct `750702271865`) | +| CH domain | `ch.sorobanscan.rumblefish.dev` | +| Prod CH shell | `ssh -i ~/.ssh/sorban-prod_ed25519 deploy@168.119.73.161` → `docker exec -i app-clickhouse-1 clickhouse-client --query='…'` | +| Activation ledger | `50457424` (split between `combined` and `sdex-only`) | +| Combined floor (`--end` max) | `63352611` (live starts at `63352612`) | +| Chunk size | 5 × 64,000 = 320,000 ledgers (bump/shrink by whole partitions) | +| Measured pace | ~184 KB/ledger, ~64 min/100k on a home line — download-bound | + +See also: `docs/runbooks/running-ingestion-components.md` (full flag reference, +minute-seam rules) and `docs/runbooks/seed-pool-registry.md` (registry seeding). diff --git a/lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md b/lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md index 2c0dce3..14555aa 100644 --- a/lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md +++ b/lore/1-tasks/active/0053_FEATURE_soroban-amm-backfill-cli-stream-1-impl/README.md @@ -449,8 +449,10 @@ only at run-end). Chunk boundary = one accepted one-minute seam per source. | `[1, 50457423]` pre-Soroban SDEX tail | ~50.5M ledgers (~4× larger) — run `--mode sdex-only` after the combined range, with `--tip `. | **Measured throughput (2026-07-07):** ~184 KB/ledger, ~4.8 MB/s, ~64 min/100k — -~100% download-bound. **Strongly consider a us-east-1 EC2** (S3-local) to collapse -download time before committing weeks of home bandwidth. +~100% download-bound. **Strongly consider a us-east-2 EC2** (the +`aws-public-blockchain` bucket is in us-east-2 — same-region S3) to collapse +download time before committing weeks of home bandwidth. Full first-timer +runbook: `docs/runbooks/continue-soroban-backfill.md`. **`--end` floor stays 63352611** (SDEX live floor − 1). Backfill now contaminates `min(sdex ledger)` in `price_ohlcv_1m`, so re-derive the live floor from a