feat(protocol): bank the v18 per-second fields the storage funnel was discarding

[vishk23]

extractHistoricalStreams is a narrow funnel: anything it doesn't name is decoded and dropped one line later — and the strap trims its banked history the moment NOOP acks the offload, so an unbanked field is gone permanently and can never be censused or validated. #845 inventories 23 such fields.

This lands every remaining 5/MG v18 per-second field as durable capture.

Four named channels ride existing per-second rows (all nullable, no DEFAULT):

Column	Table	Field
dynAccel REAL	gravitySample	dynamic_acceleration@41
rawByte INTEGER	sleepStateSample	the whole @81 byte (state untouched = (rawByte>>4)&3)
aux1Raw / aux2Raw INTEGER	skinTempSample	temp_aux_1_raw@69 / temp_aux_2_raw@71

dynamic_acceleration goes beside the gravity vector deliberately: NOOP's motion spine derives stillness from 1 Hz gravity deltas — orientation change, not acceleration — so storing the strap's own gravity-removed magnitude next to the proxy is what makes a later comparison possible at all.

The other fifteen fields go in one narrow side table v18AuxSample(deviceId, ts, fields BLOB) rather than fifteen columns, for three reasons: (1) no existing per-second table is guaranteed present — gravitySample needs gravity_x, skinTempSample needs @73 past its thermal gate, hrSample skips bpm = 0 — so a v18 record can carry aux fields while all of those gate out, and a hung-off-column design would silently drop records; (2) it keeps fifteen unpinned bytes out of tables analytics read; (3) a new slot is a bitmap bit, not a migration. Wire format: 1 version byte + u16 presence bitmap + only the present slots, LE, ~30 B full. Byte-identical Swift/Kotlin codecs, and both suites pin the same fixture hex, so a one-sided edit fails a test rather than writing blobs the other platform can't read from a .noopbak.

Slots use the decoder's own names verbatim — several are frankly unpinned bytes and nothing is renamed to imply a meaning it hasn't earned: record_index@11, rr_count@23, cardiac_flags@33, hr_fixed_8_8@36, rr_packed@38, cardiac_status@40, step_cadence@59, status_word@75, status_word_1@77, status_word_2@79, aux_byte_82@82 (raw; upstream's gated spo2_candidate_82 untouched, nothing promoted to spo2Pct), optical_baseline_106@106, optical_amp_a@108, optical_amp_b@109, unknown_f32_113@113 (banked as its raw u32 bit pattern so a future census can reinterpret the bytes).

Two fields deliberately not banked twice: motion_wear_quality@63 is the same byte as activity_class under a second name, and spo2_candidate_82 is derivable from the raw @82. A test enforces that no slot duplicates an already-persisted field.

Instrumentation only. No analytic, score, gate, UI or export reads any of it.

WHOOP 4.0 is byte-identical — the aux gate keys on hist_version == 18, not on which keys happen to be present. That's load-bearing: rr_count is shared with the 4.0 schema, so a presence test would have banked a near-empty row for every 4.0 second.

Migrations: GRDB v31-deep-capture-channels + Room MIGRATION_24_25 (db 24→25), additive ALTERs only, nothing renamed or reordered. Migration tests both sides; the Kotlin one asserts the hand-written SQL is ADD COLUMN with no NOT NULL, no DEFAULT, and no DROP/DELETE/UPDATE/INSERT.

Storage: ~75 B per v18 strap-second ≈ 2 MB per 8 h night. For scale, ppgWaveformSample already banks ~50 B per v26-second. No retention cap — matching every other durable decoded stream — but these tables are added to the storage-stats readout so the growth is visible.

Second commit (fix(store)) documents ppgWaveformSample's intended write-only status at its migration (it's instrumentation under the post-#194 rule, so it shouldn't be "cleaned up"), and fixes two gaps that make write-only data hazardous: ppgWaveformSample, rawImuSample, sleepStateSample and v18AuxSample were all missing from the #547 timestamp heal despite sharing the same ts and ingest path, and all four were missing from storageStats() — so unbounded unpruned tables were invisible in the only space readout users have.

Tests: WhoopProtocol 348 (11 new), WhoopStore 311 (14 new), StrandAnalytics 1150, NoopLocalAccess 9 — all 0 failures. Android testFullDebugUnitTest 3084 passing, 30 new; one pre-existing unrelated failure (SyncChipStateTest) reproduced on pristine upstream/main. macOS Strand BUILD SUCCEEDED.

[ryanbr]

Strong PR. The blob-with-presence-bitmap is the right shape, "absence is first-class, never 0" is the correct call for unpinned bytes, both suites pinning the same fixture hex is exactly the parity discipline this needs, and gating the aux row on hist_version == 18 rather than key-presence is a subtle catch — rr_count really would have banked a near-empty row for every 4.0 second. Migration numbering is clean against current main (GRDB v30 / Room 24 after #830, so v31 / 24→25 is right).

One blocker: this banks a field under a name your own census disproved 25 minutes earlier.

Streams.swift:307:

case hrFixed88   // @36  u16 — higher-precision HR: bpm = value/256

Your #845 analysis showed u16@36/256 is (flags << 8 | hr) / 256 — the integer part IS @37, and the "fraction" is the flag byte over 256. I verified it against the doc's own worked example: 25997 = frame[37]=101, frame[36]=141, so 101.55 is literally 101 + 141/256. There is no sub-bpm HR in v18.

The data is preserved either way, since a u16 keeps both bytes. The problem is the name and comment become a storage contract — V18AuxSlot.rawValue is the wire ordering, so this is the expensive kind of thing to correct later. It also contradicts the PR's own stated principle that "nothing is renamed to imply a meaning it hasn't earned". Suggest splitting into flagsAt36 (u8) + hrDupAt37 (u8), or keeping the u16 under a neutral name like rawU16At36 — before this ships, not after.

Same class, smaller: opticalAmpA's comment says 128 = invalid sentinel, but your census found it is record-level — amp_a == 128 and amp_b == 128 co-occur 757/757 and never independently. Worth wording as one record marker.

One thing to correct in the description: "No retention cap — matching every other durable decoded stream" is not accurate. rawImuSample caps at 3600 rows (rawImuRetentionRows, #423) — and it is the closest analogue here, raw instrumentation banked as a blob. At ~2 MB/night unbounded this is ~730 MB/year on a device with no server to offload to. Adding these tables to the storage readout is good, but visible-and-unbounded is still unbounded. Given nothing reads these yet, a rolling cap seems easier to relax later than to impose after users have a year of rows.

Not blocking, worth noting: the second commit's timestamp-heal fix for ppgWaveformSample / rawImuSample / sleepStateSample is a genuine pre-existing bug fix that stands on its own. The storageStats half clearly belongs here; the heal half would be easier to review — and to revert — separately.

[ryanbr]

Re-reviewed the codecs properly this time. The Swift unpack is solid — truncation-guarded, no over-read, unknown-version returns empty, trailing bytes ignored so a newer writer stays readable. Kotlin masks with and 0xFF throughout, so no sign-extension bug on individual bytes.

But the two 4-byte slots decode to different NUMBERS on the two platforms, and your parity tests bake it in.

Swift Int is 64-bit; Kotlin Int is 32-bit. For a u32 with bit 31 set:

Swift  Int: 3_232_604_573      Kotlin Int: -1_062_362_723

Your own fixtures use exactly such a value, and each suite asserts its own platform's answer:

unknownF32Bits: 0xC0A7_619D                      // Swift: positive

assertEquals(0xC0A7619D.toInt(), a.unknownF32Bits)   // Kotlin: negative

Both pass. The fixture-hex tests can't catch it either, because the wire bytes are identical — the divergence is in the decoded value, downstream of the bytes you pinned. The unknownF32At113 float accessor is fine on both (same bit pattern → −5.2307), so this only bites anything reading the raw integer.

It is not an edge case for @113: your census says it floors at −5.2869 when quality is good, so bit 31 is set in a large share of records. recordIndex@11 is also u32 — safe at 1 Hz for ~68 years unless the strap's counter does not start near zero.

Suggestion, minimal and wire-compatible:

// V18AuxCodec: widen the decoded domain to match Swift's 64-bit Int.
val values = arrayOfNulls<Long>(SLOTS.size)
var u = 0L
for (b in 0 until slot.width) u = u or ((data[i + b].toLong() and 0xFF) shl (8 * b))

with slotValues: List<Long?>, then assert the same literal on both sides — 3_232_604_573 — so the test proves parity instead of documenting its absence. Blob bytes are unchanged, so no migration and no re-write of existing rows.

Combined with the hrFixed88 naming point above, both are cheap now and expensive once rows exist.