feat(scores): add hybrid v2 score rollout

.env.example

@@ -3,3 +3,10 @@
 
 NUXT_PUBLIC_NIMIQ_NETWORK=test-albatross
 ALBATROSS_RPC_NODE_URL=
+# off: v1 only; shadow: write v1/v2, serve v1; active: write v1/v2, serve v2
+NUXT_SCORE_V2_MODE=off
+
+# Migration-only NuxtHub D1 HTTP credentials. Keep out of deployed runtime vars.
+NUXT_HUB_CLOUDFLARE_ACCOUNT_ID=
+NUXT_HUB_CLOUDFLARE_DATABASE_ID=
+NUXT_HUB_CLOUDFLARE_API_TOKEN=

MIGRATION.md

@@ -2,7 +2,7 @@
 
 ## Background
 
-Cloudflare Pages does NOT support scheduled tasks (cron jobs). This project requires hourly syncing, so we migrated to Cloudflare Workers.
+Cloudflare Pages does not support scheduled tasks. This project requires six-hour syncing, so it uses Cloudflare Workers.
 
 ## Setting Up Redirects
 
@@ -60,3 +60,72 @@ curl -I https://validators-api-testnet.pages.dev/api/v1/status
 2. Deploy redirects to Pages projects
 3. Monitor for 1-2 weeks
 4. (Optional) Deprecate legacy `.pages.dev` URLs and add a custom domain
+
+## Activity integrity and score v2 rollout
+
+This implementation does not perform any remote migration or deployment. Run each remote command manually, testnet first, after reviewing its target.
+
+### Safeguards
+
+1. Authenticate Wrangler and confirm the expected account:
+
+   ```bash
+   pnpm exec wrangler login
+   pnpm exec wrangler whoami
+   ```
+
+2. Inspect applied migrations and relevant table/index definitions. Never assume a migration is already applied:
+
+   ```bash
+   pnpm exec wrangler d1 execute validators-api-testnet --remote --env testnet --command "SELECT id, name, applied_at FROM _hub_migrations ORDER BY id;"
+   pnpm exec wrangler d1 execute validators-api-testnet --remote --env testnet --command "SELECT name, type, sql FROM sqlite_schema WHERE name IN ('_hub_migrations', 'activities', 'activity_epochs', 'scores', 'validators') OR name LIKE 'idx_%' ORDER BY type, name;"
+   ```
+
+3. Export a backup outside this repository:
+
+   ```bash
+   pnpm exec wrangler d1 export validators-api-testnet --remote --env testnet --output ../validators-api-testnet-before-score-v2.sql
+   ```
+
+4. Put migration-only D1 HTTP credentials in `.env.testnet`:
+
+   ```dotenv
+   NUXT_HUB_CLOUDFLARE_ACCOUNT_ID=...
+   NUXT_HUB_CLOUDFLARE_DATABASE_ID=...
+   NUXT_HUB_CLOUDFLARE_API_TOKEN=...
+   ```
+
+   Keep these values out of deployed runtime variables. Migration scripts enable the HTTP driver only for their own command and fail if any credential is missing.
+
+Repeat inspection and backup with `validators-api-mainnet` and without `--env testnet` only after testnet validation passes.
+
+### Testnet-first sequence
+
+1. Keep testnet `NUXT_SCORE_V2_MODE=off`.
+2. Apply all pending tracked migrations:
+
+   ```bash
+   pnpm db:migrate:testnet
+   ```
+
+   This uses NuxtHub's basename-compatible `_hub_migrations` ledger. Do not substitute `wrangler d1 migrations apply`; Wrangler records full filenames and can replay an existing NuxtHub baseline.
+
+3. Inspect `_hub_migrations` and relevant schemas again.
+4. Deploy testnet while v2 remains off.
+5. Let the six-hour job discover epochs, repair recent activity, store the snapshot, then calculate v1 scores.
+6. Set `NUXT_SCORE_V2_MODE=shadow`, deploy, and validate v1/v2 rows, activity coverage, score versions, and `current`, `stale`, or `no_score` API states.
+7. Set `NUXT_SCORE_V2_MODE=active` only after shadow results pass.
+8. Repeat the same inspect, backup, migrate, off, repair, shadow, validate, and active sequence for mainnet.
+
+Activity marker interpretation during validation:
+
+- `syncing`: repair attempt is inside its six-hour lease.
+- `complete`: stored `finalized` marker has an exact elected set and matching counts.
+- `incomplete`: expected activity has no valid finalized marker.
+- `failed`: attempt rolled back and recorded an error for retry.
+
+API list, detail, and status endpoints accept `score-version=1|2`. Omitted version follows rollout mode: `off` and `shadow` select v1; `active` selects v2. Selected-version rows are filtered before latest-score selection. `current` means latest completed epoch is covered, `stale` preserves an older valid score, and `no_score` returns null score values.
+
+### Rollback
+
+Set `NUXT_SCORE_V2_MODE=off`, deploy the configuration change, and request `score-version=1` explicitly while caches settle. Keep both v1 and v2 rows intact. Do not roll back schema or score data destructively.

README.md

@@ -113,8 +113,19 @@ The Validators API provides endpoints to retrieve validator information for inte
 | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
 | [/api/v1/validators](https://validators-api-main.je-cf9.workers.dev/api/v1/validators) | Retrieves the validator list. See [query params](./server/utils/schemas.ts#L54) |
 | [/api/v1/validators/:validator_address](https://validators-api-main.je-cf9.workers.dev/api/v1/validators/NQ98%20D3KE%208EQ8%20Y7DK%20G1MT%203P5T%202PHX%2018V5%20UEC1) | Retrieves the validator information |
+| [/api/v1/status](https://validators-api-main.je-cf9.workers.dev/api/v1/status) | Retrieves activity coverage and selected score status |
 | [/api/v1/supply](https://validators-api-main.je-cf9.workers.dev/api/v1/supply) | Retrieves supply status |
 
+Validator list, detail, and status endpoints accept `score-version=1|2`. Without it, `NUXT_SCORE_V2_MODE=off` and `shadow` select v1, while `active` selects v2. Queries filter by the selected version before choosing the latest score, so responses never mix v1 and v2 history.
+
+For score v2, finalized gaps after a validator has been elected estimate the chance of receiving zero of the 512 validator slots from the nearest surrounding stake observations. A complete gap is marked `inferred_offline`, shown in v2 validator activity, and penalized by v2 only when estimated stake is at least 1% and random-election probability is below 0.1%. Lower-stake and lower-confidence gaps remain `not_elected_randomness` and do not reduce availability. Score v1 behavior remains unchanged.
+
+Every score includes its version and data state:
+
+- `current`: selected-version score covers the latest completed epoch.
+- `stale`: an older valid score remains available while current activity or scoring is incomplete.
+- `no_score`: no selected-version score exists; numeric fields remain `null`, not zero.
+
 ## Validators Dashboard
 
 The Validators Dashboard is a simple Nuxt application that displays all validators along with their scores. You can access the dashboard here: https://validators-api-main.je-cf9.workers.dev/
@@ -138,7 +149,18 @@ We also do have an UI component to visualize the range, check the status, and de
 
 ### Fetcher
 
-The fetcher is a process that retrieves data from the Nimiq network and stores it in a D1 database. The fetcher runs every hour and collects data about the validators in two different ways:
+The fetcher retrieves data from the Nimiq network and stores it in D1. It runs every six hours in this order:
+
+1. Discover completed epochs and verify or repair activity snapshots.
+2. Store the current validator snapshot.
+3. Calculate scores from finalized activity markers.
+
+Completed-epoch activity uses marker-backed integrity checks. Operator-facing marker states are:
+
+- `syncing`: repair attempt owns a fresh six-hour lease.
+- `complete`: stored `finalized` marker has matching election-set hash and elected counts.
+- `incomplete`: expected epoch has no finalized marker or exact stored set.
+- `failed`: attempt rolled back and retained its error for retry.
 
 #### Ended epochs
 
@@ -213,44 +235,35 @@ Where `env`: `testnet` (omit `-e env` for mainnet production).
 
 ### D1 Migrations
 
-When adding a new SQL migration under `server/db/migrations/`, apply it to the remote D1 database.
-
-For the `cron_runs` table:
-
-```bash
-pnpm db:apply:cron-runs:mainnet
-```
+Never assume remote migration state. Authenticate Wrangler, inspect `_hub_migrations` and relevant schemas, and export a backup outside the repository before applying migrations. Set the migration-only `NUXT_HUB_CLOUDFLARE_ACCOUNT_ID`, `NUXT_HUB_CLOUDFLARE_DATABASE_ID`, and `NUXT_HUB_CLOUDFLARE_API_TOKEN` values in the target `.env` file.
 
-Testnet:
+Generic migration commands use NuxtHub's `_hub_migrations` basenames and apply all pending files under `server/db/migrations/`:
 
 ```bash
-pnpm db:apply:cron-runs:testnet
+pnpm db:migrate:testnet
+pnpm db:migrate:mainnet
 ```
 
-Required schema:
+Always migrate and validate testnet first. See [MIGRATION.md](./MIGRATION.md) for inspection, backup, rollout, and rollback steps.
 
-- `validators.is_listed` must exist in all remote D1 databases.
+This implementation does not run any remote migration or deployment.
 
-If the column is missing, apply it manually:
+**Environments** (configured in `wrangler.json`):
 
-Mainnet:
+- `production`: [Validators API Mainnet](https://validators-api-main.je-cf9.workers.dev) via manual `wrangler deploy`
+- `testnet`: [Validators API Testnet](https://validators-api-test.je-cf9.workers.dev) via manual `wrangler deploy --env testnet`
 
-```bash
-pnpm db:apply:is-listed:mainnet
-```
+Each environment has its own D1 database, KV cache, and R2 blob. Sync runs every six hours via Cloudflare cron triggers (see `server/tasks/sync/`).
 
-Testnet:
+### Score v2 rollout
 
-```bash
-pnpm db:apply:is-listed:testnet
-```
-
-**Environments** (configured in `wrangler.json`):
+Set `NUXT_SCORE_V2_MODE` per environment:
 
-- `production`: [Validators API Mainnet](https://validators-api-main.je-cf9.workers.dev) via manual `wrangler deploy`
-- `testnet`: [Validators API Testnet](https://validators-api-test.je-cf9.workers.dev) via manual `wrangler deploy --env testnet`
+- `off`: write and serve v1 only.
+- `shadow`: keep v1 writes, also write v2, and serve v1 by default.
+- `active`: keep v1 and v2 writes, and serve v2 by default.
 
-Each environment has its own D1 database, KV cache, and R2 blob. Sync runs every 12 hours via Cloudflare cron triggers (see `server/tasks/sync/`).
+Rollback requires no destructive schema or data change: set `NUXT_SCORE_V2_MODE=off`, keep all v1/v2 rows intact, and request `score-version=1` explicitly while the configuration change propagates.
 
 ### Deployment Migration
 

app/app.vue

@@ -1,17 +1,93 @@
 <script setup lang="ts">
 import type { EnvItemType } from './utils/environments'
 import { environments, getEnvironmentItem } from './utils/environments'
+import { mergeScoreVersionQuery, resolveDashboardScoreVersion } from './utils/score-version'
 
-const { data: status, status: statusRequest, refresh: refreshStatus, error } = await useFetch('/api/v1/status', { server: true, lazy: false })
+const route = useRoute()
+const { scoreVersionRequestQuery, setScoreVersion } = useScoreVersionQuery()
+const { data: status, status: statusRequest, refresh: refreshStatus, error } = await useFetch('/api/v1/status', {
+  server: true,
+  lazy: false,
+  query: scoreVersionRequestQuery,
+})
+const selectedScoreVersion = computed(() =>
+  resolveDashboardScoreVersion(
+    route.query['score-version'],
+    status.value?.selectedScoreVersion,
+  ),
+)
+const scoreVersionLinkQuery = computed(() =>
+  mergeScoreVersionQuery({}, selectedScoreVersion.value),
+)
 
 const colorMode = useColorMode()
 const toggleDark = () => colorMode.value = colorMode.value === 'light' ? 'dark' : 'light'
 
-const route = useRoute()
 const validatorDetail = computed(() => !!route.params.address)
-const isActivitySync = computed(() => Boolean(status.value?.missingEpochs?.length === 0))
-const isScoreSync = computed(() => status.value?.missingScore === false)
-const isSynced = computed(() => isActivitySync.value && isScoreSync.value)
+
+type HealthKind = 'current' | 'synchronizing' | 'failed' | 'stale' | 'no_score' | 'live_unavailable'
+
+const health = computed<{ kind: HealthKind, label: string }>(() => {
+  if (statusRequest.value === 'pending')
+    return { kind: 'synchronizing', label: 'Synchronizing' }
+  if (error.value || status.value?.failedEpochs.length)
+    return { kind: 'failed', label: 'Failed' }
+  if (status.value?.syncingEpochs.length)
+    return { kind: 'synchronizing', label: 'Synchronizing' }
+  if (!status.value?.range)
+    return { kind: 'live_unavailable', label: 'Live unavailable' }
+  if (status.value.scoreStatus === 'no_score')
+    return { kind: 'no_score', label: 'No score' }
+  if (
+    status.value.recentCoverage !== 1
+    || status.value.longTermCoverage !== 1
+    || status.value.scoreStatus === 'stale'
+  ) {
+    return { kind: 'stale', label: 'Stale' }
+  }
+  return { kind: 'current', label: 'Current' }
+})
+
+const healthClasses = computed(() => {
+  if (health.value.kind === 'current') {
+    return {
+      container: 'outline-green-500 text-green-1100',
+      badge: 'bg-green-400',
+    }
+  }
+  if (health.value.kind === 'synchronizing' || health.value.kind === 'live_unavailable') {
+    return {
+      container: 'outline-neutral-400 text-neutral-800',
+      badge: 'bg-neutral-400',
+    }
+  }
+  return {
+    container: 'outline-red-500 text-red-1100',
+    badge: 'bg-red-400',
+  }
+})
+
+const healthDetails = computed(() => {
+  const details: string[] = []
+  if (error.value)
+    details.push('Status request failed.')
+  if (!status.value)
+    return details
+  if (!status.value.range)
+    details.push('Live blockchain range is unavailable. Stored synchronization state is shown.')
+  if (status.value.failedEpochs.length)
+    details.push(`Failed activity epochs: ${status.value.failedEpochs.map(epoch => epoch.epochNumber).join(', ')}.`)
+  if (status.value.syncingEpochs.length)
+    details.push(`Synchronizing activity epochs: ${status.value.syncingEpochs.map(epoch => epoch.epochNumber).join(', ')}.`)
+  details.push(`Recent activity coverage: ${status.value.recentCoverage === null ? 'unavailable' : percentageFormatter.format(status.value.recentCoverage)}.`)
+  details.push(`Long-term activity coverage: ${status.value.longTermCoverage === null ? 'unavailable' : percentageFormatter.format(status.value.longTermCoverage)}.`)
+  details.push(`Score v${status.value.selectedScoreVersion}: ${status.value.scoreStatus.replace('_', ' ')}.`)
+  return details
+})
+
+const showHealthWarning = computed(() =>
+  statusRequest.value !== 'pending' && (health.value.kind !== 'current' || Boolean(error.value)),
+)
 
 const { nimiqNetwork } = useSafeRuntimeConfig().public
 const [DefineEnvItem, EnvItem] = createReusableTemplate<{ item: EnvItemType, component: string }>()
@@ -30,16 +106,34 @@ const currentEnvItem = getEnvironmentItem(nimiqNetwork) ?? { network: nimiqNetwo
   </DefineEnvItem>
 
   <div flex="~ col gap-64" mx-auto size-screen max-h-screen max-w-1200 px-32 py-20>
-    <header flex="~ gap-32 row items-center">
-      <NuxtLink to="/" flex>
+    <header flex="~ gap-32 row items-center" class="max-sm:flex-wrap max-sm:gap-12">
+      <NuxtLink
+        :to="{
+          path: '/',
+          query: scoreVersionLinkQuery,
+        }"
+        flex
+      >
         <div aria-hidden class="i-nimiq:logos-nimiq-horizontal dark:i-nimiq:logos-nimiq-white-horizontal !ml-16 !h-24 !w-90" />
         <span ml-8 text-16 font-light tracking-0.75>Validators</span>
       </NuxtLink>
-      <NuxtLink v-if="validatorDetail" to="/" block w-max nq-arrow-back nq-ghost-btn>
+      <NuxtLink
+        v-if="validatorDetail"
+        :to="{
+          path: '/',
+          query: scoreVersionLinkQuery,
+        }"
+        block w-max nq-arrow-back nq-ghost-btn
+      >
         Go back
       </NuxtLink>
-      <div ml-auto>
-        <div flex="~ items-center gap-8" outline="~ 1.5" :class="statusRequest === 'pending' ? 'outline-neutral/10 text-neutral-800' : isSynced ? 'outline-green-500 text-green-1100' : 'outline-red-500 text-red-1100'" rounded-6 f-text-2xs font-semibold of-clip>
+      <div ml-auto flex="~ items-center gap-12" class="max-sm:order-2 max-sm:ml-0 max-sm:w-full max-sm:justify-between">
+        <ScoreVersionSelect
+          :model-value="selectedScoreVersion"
+          :disabled="statusRequest === 'pending'"
+          @update:model-value="setScoreVersion"
+        />
+        <div flex="~ items-center gap-8" outline="~ 1.5" :class="healthClasses.container" rounded-6 f-text-2xs font-semibold of-clip>
           <CollapsibleRoot w-full>
             <CollapsibleTrigger bg-transparent w-full relative group rounded="6 reka-open:b-0" transition-border-radius of-clip>
               <EnvItem :item="currentEnvItem" component="div" />
@@ -53,18 +147,18 @@ const currentEnvItem = getEnvironmentItem(nimiqNetwork) ?? { network: nimiqNetwo
             </CollapsibleContent>
           </CollapsibleRoot>
 
-          <div flex="~ items-center gap-8" f-px-2xs py-6 whitespace-nowrap :title="`Status for nimiq+${nimiqNetwork}`" :class="statusRequest === 'pending' ? 'bg-neutral-400' : isSynced ? 'bg-green-400' : 'bg-red-400'">
-            <template v-if="statusRequest === 'pending'">
+          <div flex="~ items-center gap-8" f-px-2xs py-6 whitespace-nowrap :title="`Status for nimiq+${nimiqNetwork}`" :class="healthClasses.badge">
+            <template v-if="health.kind === 'synchronizing'">
               <div class="i-nimiq:spinner" />
-              Getting health
+              {{ health.label }}
             </template>
-            <template v-else-if="isSynced">
+            <template v-else-if="health.kind === 'current'">
               <div i-nimiq:duotone-fluctuations f-text-xl />
-              API synced
+              {{ health.label }}
             </template>
             <template v-else>
               <div i-nimiq:alert op-70 f-text-xs />
-              Error
+              {{ health.label }}
             </template>
           </div>
         </div>
@@ -73,19 +167,19 @@ const currentEnvItem = getEnvironmentItem(nimiqNetwork) ?? { network: nimiqNetwo
       <button class="i-nimiq:moon" @click="() => toggleDark()" />
     </header>
     <main flex-1>
-      <div v-if="(!isSynced || error) && $route.path === '/'" bg="red/8" outline="1.5 ~ red-600" rounded-12 f-p-md text="14 red-1100" nq-prose-compact children:max-w-none f-mb-lg>
+      <div v-if="showHealthWarning && $route.path === '/'" bg="red/8" outline="1.5 ~ red-600" rounded-12 f-p-md text="14 red-1100" nq-prose-compact children:max-w-none f-mb-lg>
         <h1 flex="~ items-center gap-12" text-red-1100 f-text-lg>
           <div i-nimiq:alert op-70 text-0.9em m-0 />
-          <template v-if="!isActivitySync">
-            Activity out of sync
-          </template>
-          <template v-else-if="!isScoreSync">
-            Score not computed
-          </template>
+          {{ health.label }}
         </h1>
         <p f-mt-2xs>
-          The database is not fully synchronized with the blockchain. The API may not return the most recent data.
+          API data is not fully current. Stored data remains available where possible.
         </p>
+        <ul v-if="healthDetails.length" f-mt-xs>
+          <li v-for="detail in healthDetails" :key="detail">
+            {{ detail }}
+          </li>
+        </ul>
 
         <pre v-if="error" bg="red/6" text="f-2xs red-1100" outline="red/30" w-inherit>{{ JSON.stringify(error, null, 2) }}</pre>
 
@@ -113,7 +207,7 @@ const currentEnvItem = getEnvironmentItem(nimiqNetwork) ?? { network: nimiqNetwo
         <hr f-my-sm border-red-600>
 
         <p f-mt-md text="f-sm red-1100/80">
-          <strong>Note:</strong> Data synchronization is handled automatically by scheduled tasks that run every 12 hours. A score lag of up to 1 epoch can be expected between sync cycles.
+          <strong>Note:</strong> Data synchronization is handled automatically by scheduled tasks that run every six hours. A score lag of up to 1 epoch can be expected between sync cycles.
         </p>
       </div>