diff --git a/.env.example b/.env.example index 0e0e752..3239b59 100644 --- a/.env.example +++ b/.env.example @@ -3,3 +3,10 @@ NUXT_PUBLIC_NIMIQ_NETWORK=test-albatross ALBATROSS_RPC_NODE_URL= +# off: v1 only; shadow: write v1/v2, serve v1; active: write v1/v2, serve v2 +NUXT_SCORE_V2_MODE=off + +# Migration-only NuxtHub D1 HTTP credentials. Keep out of deployed runtime vars. +NUXT_HUB_CLOUDFLARE_ACCOUNT_ID= +NUXT_HUB_CLOUDFLARE_DATABASE_ID= +NUXT_HUB_CLOUDFLARE_API_TOKEN= diff --git a/MIGRATION.md b/MIGRATION.md index 00546a9..f996e09 100644 --- a/MIGRATION.md +++ b/MIGRATION.md @@ -2,7 +2,7 @@ ## Background -Cloudflare Pages does NOT support scheduled tasks (cron jobs). This project requires hourly syncing, so we migrated to Cloudflare Workers. +Cloudflare Pages does not support scheduled tasks. This project requires six-hour syncing, so it uses Cloudflare Workers. ## Setting Up Redirects @@ -60,3 +60,72 @@ curl -I https://validators-api-testnet.pages.dev/api/v1/status 2. Deploy redirects to Pages projects 3. Monitor for 1-2 weeks 4. (Optional) Deprecate legacy `.pages.dev` URLs and add a custom domain + +## Activity integrity and score v2 rollout + +This implementation does not perform any remote migration or deployment. Run each remote command manually, testnet first, after reviewing its target. + +### Safeguards + +1. Authenticate Wrangler and confirm the expected account: + + ```bash + pnpm exec wrangler login + pnpm exec wrangler whoami + ``` + +2. Inspect applied migrations and relevant table/index definitions. Never assume a migration is already applied: + + ```bash + pnpm exec wrangler d1 execute validators-api-testnet --remote --env testnet --command "SELECT id, name, applied_at FROM _hub_migrations ORDER BY id;" + pnpm exec wrangler d1 execute validators-api-testnet --remote --env testnet --command "SELECT name, type, sql FROM sqlite_schema WHERE name IN ('_hub_migrations', 'activities', 'activity_epochs', 'scores', 'validators') OR name LIKE 'idx_%' ORDER BY type, name;" + ``` + +3. Export a backup outside this repository: + + ```bash + pnpm exec wrangler d1 export validators-api-testnet --remote --env testnet --output ../validators-api-testnet-before-score-v2.sql + ``` + +4. Put migration-only D1 HTTP credentials in `.env.testnet`: + + ```dotenv + NUXT_HUB_CLOUDFLARE_ACCOUNT_ID=... + NUXT_HUB_CLOUDFLARE_DATABASE_ID=... + NUXT_HUB_CLOUDFLARE_API_TOKEN=... + ``` + + Keep these values out of deployed runtime variables. Migration scripts enable the HTTP driver only for their own command and fail if any credential is missing. + +Repeat inspection and backup with `validators-api-mainnet` and without `--env testnet` only after testnet validation passes. + +### Testnet-first sequence + +1. Keep testnet `NUXT_SCORE_V2_MODE=off`. +2. Apply all pending tracked migrations: + + ```bash + pnpm db:migrate:testnet + ``` + + This uses NuxtHub's basename-compatible `_hub_migrations` ledger. Do not substitute `wrangler d1 migrations apply`; Wrangler records full filenames and can replay an existing NuxtHub baseline. + +3. Inspect `_hub_migrations` and relevant schemas again. +4. Deploy testnet while v2 remains off. +5. Let the six-hour job discover epochs, repair recent activity, store the snapshot, then calculate v1 scores. +6. Set `NUXT_SCORE_V2_MODE=shadow`, deploy, and validate v1/v2 rows, activity coverage, score versions, and `current`, `stale`, or `no_score` API states. +7. Set `NUXT_SCORE_V2_MODE=active` only after shadow results pass. +8. Repeat the same inspect, backup, migrate, off, repair, shadow, validate, and active sequence for mainnet. + +Activity marker interpretation during validation: + +- `syncing`: repair attempt is inside its six-hour lease. +- `complete`: stored `finalized` marker has an exact elected set and matching counts. +- `incomplete`: expected activity has no valid finalized marker. +- `failed`: attempt rolled back and recorded an error for retry. + +API list, detail, and status endpoints accept `score-version=1|2`. Omitted version follows rollout mode: `off` and `shadow` select v1; `active` selects v2. Selected-version rows are filtered before latest-score selection. `current` means latest completed epoch is covered, `stale` preserves an older valid score, and `no_score` returns null score values. + +### Rollback + +Set `NUXT_SCORE_V2_MODE=off`, deploy the configuration change, and request `score-version=1` explicitly while caches settle. Keep both v1 and v2 rows intact. Do not roll back schema or score data destructively. diff --git a/README.md b/README.md index 93d47c8..c9f19eb 100644 --- a/README.md +++ b/README.md @@ -113,8 +113,19 @@ The Validators API provides endpoints to retrieve validator information for inte | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | [/api/v1/validators](https://validators-api-main.je-cf9.workers.dev/api/v1/validators) | Retrieves the validator list. See [query params](./server/utils/schemas.ts#L54) | | [/api/v1/validators/:validator_address](https://validators-api-main.je-cf9.workers.dev/api/v1/validators/NQ98%20D3KE%208EQ8%20Y7DK%20G1MT%203P5T%202PHX%2018V5%20UEC1) | Retrieves the validator information | +| [/api/v1/status](https://validators-api-main.je-cf9.workers.dev/api/v1/status) | Retrieves activity coverage and selected score status | | [/api/v1/supply](https://validators-api-main.je-cf9.workers.dev/api/v1/supply) | Retrieves supply status | +Validator list, detail, and status endpoints accept `score-version=1|2`. Without it, `NUXT_SCORE_V2_MODE=off` and `shadow` select v1, while `active` selects v2. Queries filter by the selected version before choosing the latest score, so responses never mix v1 and v2 history. + +For score v2, finalized gaps after a validator has been elected estimate the chance of receiving zero of the 512 validator slots from the nearest surrounding stake observations. A complete gap is marked `inferred_offline`, shown in v2 validator activity, and penalized by v2 only when estimated stake is at least 1% and random-election probability is below 0.1%. Lower-stake and lower-confidence gaps remain `not_elected_randomness` and do not reduce availability. Score v1 behavior remains unchanged. + +Every score includes its version and data state: + +- `current`: selected-version score covers the latest completed epoch. +- `stale`: an older valid score remains available while current activity or scoring is incomplete. +- `no_score`: no selected-version score exists; numeric fields remain `null`, not zero. + ## Validators Dashboard The Validators Dashboard is a simple Nuxt application that displays all validators along with their scores. You can access the dashboard here: https://validators-api-main.je-cf9.workers.dev/ @@ -138,7 +149,18 @@ We also do have an UI component to visualize the range, check the status, and de ### Fetcher -The fetcher is a process that retrieves data from the Nimiq network and stores it in a D1 database. The fetcher runs every hour and collects data about the validators in two different ways: +The fetcher retrieves data from the Nimiq network and stores it in D1. It runs every six hours in this order: + +1. Discover completed epochs and verify or repair activity snapshots. +2. Store the current validator snapshot. +3. Calculate scores from finalized activity markers. + +Completed-epoch activity uses marker-backed integrity checks. Operator-facing marker states are: + +- `syncing`: repair attempt owns a fresh six-hour lease. +- `complete`: stored `finalized` marker has matching election-set hash and elected counts. +- `incomplete`: expected epoch has no finalized marker or exact stored set. +- `failed`: attempt rolled back and retained its error for retry. #### Ended epochs @@ -213,44 +235,35 @@ Where `env`: `testnet` (omit `-e env` for mainnet production). ### D1 Migrations -When adding a new SQL migration under `server/db/migrations/`, apply it to the remote D1 database. - -For the `cron_runs` table: - -```bash -pnpm db:apply:cron-runs:mainnet -``` +Never assume remote migration state. Authenticate Wrangler, inspect `_hub_migrations` and relevant schemas, and export a backup outside the repository before applying migrations. Set the migration-only `NUXT_HUB_CLOUDFLARE_ACCOUNT_ID`, `NUXT_HUB_CLOUDFLARE_DATABASE_ID`, and `NUXT_HUB_CLOUDFLARE_API_TOKEN` values in the target `.env` file. -Testnet: +Generic migration commands use NuxtHub's `_hub_migrations` basenames and apply all pending files under `server/db/migrations/`: ```bash -pnpm db:apply:cron-runs:testnet +pnpm db:migrate:testnet +pnpm db:migrate:mainnet ``` -Required schema: +Always migrate and validate testnet first. See [MIGRATION.md](./MIGRATION.md) for inspection, backup, rollout, and rollback steps. -- `validators.is_listed` must exist in all remote D1 databases. +This implementation does not run any remote migration or deployment. -If the column is missing, apply it manually: +**Environments** (configured in `wrangler.json`): -Mainnet: +- `production`: [Validators API Mainnet](https://validators-api-main.je-cf9.workers.dev) via manual `wrangler deploy` +- `testnet`: [Validators API Testnet](https://validators-api-test.je-cf9.workers.dev) via manual `wrangler deploy --env testnet` -```bash -pnpm db:apply:is-listed:mainnet -``` +Each environment has its own D1 database, KV cache, and R2 blob. Sync runs every six hours via Cloudflare cron triggers (see `server/tasks/sync/`). -Testnet: +### Score v2 rollout -```bash -pnpm db:apply:is-listed:testnet -``` - -**Environments** (configured in `wrangler.json`): +Set `NUXT_SCORE_V2_MODE` per environment: -- `production`: [Validators API Mainnet](https://validators-api-main.je-cf9.workers.dev) via manual `wrangler deploy` -- `testnet`: [Validators API Testnet](https://validators-api-test.je-cf9.workers.dev) via manual `wrangler deploy --env testnet` +- `off`: write and serve v1 only. +- `shadow`: keep v1 writes, also write v2, and serve v1 by default. +- `active`: keep v1 and v2 writes, and serve v2 by default. -Each environment has its own D1 database, KV cache, and R2 blob. Sync runs every 12 hours via Cloudflare cron triggers (see `server/tasks/sync/`). +Rollback requires no destructive schema or data change: set `NUXT_SCORE_V2_MODE=off`, keep all v1/v2 rows intact, and request `score-version=1` explicitly while the configuration change propagates. ### Deployment Migration diff --git a/app/app.vue b/app/app.vue index f4638cd..f6ccac1 100644 --- a/app/app.vue +++ b/app/app.vue @@ -1,17 +1,93 @@ diff --git a/app/components/ScoreVersionSelect.vue b/app/components/ScoreVersionSelect.vue new file mode 100644 index 0000000..e19f9e8 --- /dev/null +++ b/app/components/ScoreVersionSelect.vue @@ -0,0 +1,35 @@ + + + diff --git a/app/components/ValidatorsTable.vue b/app/components/ValidatorsTable.vue index ca11aef..0c078be 100644 --- a/app/components/ValidatorsTable.vue +++ b/app/components/ValidatorsTable.vue @@ -1,11 +1,17 @@