tools(spo2): account for @82's duty cycle, screen for variance, and classify a strap that never emits it

[vishk23]

tools/linux-capture/validate_spo2_candidate.py (#814, extended by #819) is what decides whether
spo2_candidate_82 graduates to a real spo2Pct. Three properties of @82 make its current numbers
unreliable — one of them badly enough to explain the split evidence on #103.

All three were established on an 18,650-record v18 corpus: 18,602 of them from
@digitalerdude's public PacketLogger capture of an official-app
overnight sync (posted on #103 — thank you, this analysis is not possible without it), plus 48 from a
NOOP sync on a subscription-free strap.

---

1. @82 is duty-cycled, and the harness did not know

It is not sampled every second. Across the corpus it is nonzero in 450 records (2.41 %), carrying
24 distinct values in 1..160, and every one of those falls inside a window that repeats on a fixed
period:

Measurement	Value
Records with @82 != 0	450 / 18,650 (2.41 %)
Distinct nonzero values	24, range 1..160
Contiguous runs	15
Run lengths	every single one exactly 30 records
Run start phase	unix % 1200 == 337 on all 15 — zero variance
Outside the window	identically 0x00

So @82 is a 30-second measurement taken once every 20 minutes.

Why that breaks validation: a capture (or a sampling scheme) not aligned to that phase reads all
zeros and is indistinguishable from a strap with the feature switched off. Sub-sampling this very
corpus at one record per minute, off-phase, reproduces it exactly — 310 records, zero nonzero
@82, 0/16 windows covered. That is a plausible contributor to the contradictory cross-device
evidence this issue is stuck on, where one device correlated +0.99 and another moved opposite:
the second device may not have disagreed so much as missed the window.

What this PR changes

• detect_duty_cycle() measures the schedule per capture — period, phase, window length, window
  count, phase jitter — and classifies the capture absent / continuous / duty_cycled /
  irregular. Only duty_cycled changes how nights are aggregated, so a capture the tool cannot
  characterise is never silently reweighted. On the reference corpus it recovers
  period=1200s window=30s phase=337 windows=15 jitter=0 equal_length_windows=True, with nothing
  hard-coded.
• The phase is deliberately not a constant. The robust fact is that a periodic window exists and
  has a stable length; 337 is one strap's firmware and there is no evidence it generalises. The
  tests plant 900/111/20 and 600/42/10 schedules and require the tool to find them — a hard-coded
  phase fails there, rather than quietly mis-scoring a stranger's strap.
• The nightly aggregate is now the mean of the per-window means, not the mean of every second. A
  30-second burst is one measurement the strap took, not 30 independent ones; per-second averaging
  hands the night to whichever window the capture happened to sample most densely. (On the real
  corpus the two differ: 95.92 per-second vs 93.70 per-window.)
• Coverage is reported per night (windows sampled ÷ windows expected) and warned on below
  --min-window-coverage (default 0.5), because a night built on 2 of 21 windows is not comparable to
  one built on all 21.

  @82 duty cycle: mode=duty_cycled  nonzero=450/18602 (2.42%)  distinct_values=24
    detected: period=1200s  window=30s  phase=unix%1200∈[337,366]  windows_seen=15  jitter=0s  equal_length_windows=True
  window coverage: median=100% over 1 nights  low_coverage_nights=0

---

2. In-band-only screening is not specific

The harness screens candidate bytes by "value in 70–100". That is a numeric range, not a signature.

On a real subscription-free 5.0 strap, six offsets are in-band on a majority of records:

Offset	In-band share	Distinct values	What it actually is
@17	100 %	2	byte 2 of the u32 unix timestamp at @15 — reads 99–100 for a day and a half of wall clock
@33	90 %	4	cardiac_flags (already decoded)
@59	62 %	2	step_cadence (already decoded)
@69	100 %	3	temp_aux_1_raw (aux thermal channel)
@71	100 %	3	temp_aux_2_raw (aux thermal channel)
@107	62 %	4	undecoded, near-constant

Every one is either an already-decoded non-SpO₂ field or near-constant. A nightly SpO₂ cannot have
2 distinct values. The timestamp passing an SpO₂ screen is the clearest statement of the problem.

The specificity scan now requires an offset to show ≥5 distinct in-band values and stdev ≥0.5
before its correlation is ranked, rejected offsets are reported rather than dropped silently, and
@82 is held to the same floor as a checklist gate. On the reference corpus @82 clears it
(14 distinct, stdev 3.17) and @75 does not (1 value).

The regression test plants a 2-valued byte at @84 that correlates with the export better than
@82 does. Without the floor the flag wins the scan and the tool reports "not specific to @82" —
exactly the wrong conclusion, from a byte that cannot be a measurement.

---

3. A strap that never emits @82 is now classified, not failed

A subscription-free strap reads @82 = 0x00 on 100 % of its v18 records, including genuine deep
sleep. That device has not failed a correlation — it has no data to correlate, and counting it as a
FAIL biases the multi-device promote gate against the candidate for a reason that has nothing to do
with the candidate.

It is now classified feature_absent and counted as neither a PASS nor a FAIL:

strap-c,feature_absent,0,,,,None,absent,,,,,False
multi_device_eligible=2 all_pass=True feature_absent=1 (promote spo2Pct only if all_pass and ≥2 devices)

There is an evidence bar (≥100 records and ≥3 nominal periods of scored sleep) so that a short capture
which merely missed the window is not dressed up as an absence claim — that stays a plain FAIL. Given
finding 1, the distinction is the whole point.

---

Privacy

--postable keeps its property of carrying no raw SpO₂ values. The added columns are device
timing (duty_mode, duty_period_s, duty_window_s, duty_phase_s, window_coverage) and a
classification. The phase is in fact the useful thing to share, since it is what another owner needs
in order to align a capture.

Tests

tools/linux-capture: 178 → 194 (193 passed, 1 skipped, 0 failed) — python3 -m pytest tools/linux-capture/ -q, and python3 -m unittest discover (the runner the README documents) agrees
at 194. test_validate_spo2_candidate.py goes 16 → 32; all 16 pre-existing tests are unchanged
and still pass.

New coverage: schedule recovery at two planted non-reference schedules, refusal to claim a period
for scattered firings, aligned vs phase-misaligned coverage, per-window vs per-second aggregation, the
variance floor rejecting a better-correlating near-constant byte, the floor applied to @82 itself,
feature_absent classification and its evidence bar, and the postable block still carrying no raw
values.

Also verified end-to-end against the real corpus through the actual CLI path, not only synthetic
fixtures.

Scope

Python capture tooling plus the two doc sections that enumerate the harness's gates. No app code, no
decoder, no schema, no Swift/Kotlin — spo2_candidate_82 remains instrumentation-only and this does
not move it.

Serves the #103 promote gate directly: spo2_candidate_82 → spo2Pct needs ≥2 devices that each
PASS, and until now a device could fail for having captured the wrong 30 seconds out of every 20
minutes. Related: #845 (decoded-but-unread v18 fields — @82 is one of them).

Corpus credit again to @digitalerdude (#103).

[ryanbr]

Findings 1 and 2 hold up under independent testing. Finding 3 has a gap that undercuts finding 1, and I think it should be closed before this merges — it is small.

What I verified

• Suite goes 178 → 194 and the validator file 16 → 32, all green on both branches (unittest discover; pytest is not installed here).
• detect_duty_cycle really is measuring, not assuming. I planted four schedules the PR never uses — including phase > period and a 2-minute period — and it recovered every one exactly. 337 and the corpus counts appear only in comments.
• It refuses when it should: 2 windows, scattered firings, and phase jitter all fall through to irregular, and the jitter tolerance scales with capture cadence rather than being fixed. Robust to 3 of 10 windows missing without the period collapsing to a multiple.
• --postable genuinely carries no raw values. Ran it with distinctive per-night values 91/93/95/97/92/98; none appear. The new columns are timing only. (Note the new test asserts this on a flat-zero device, which cannot leak anyway — the older test at :213 is the one carrying the weight.)

The gap: feature_absent fires on capture aliasing, not just an absent feature

Finding 1 establishes that an off-phase capture reads all zeros and is indistinguishable from a strap with the feature switched off. Finding 3 then treats all-zero as evidence the feature is absent.

Same strap, same firmware, @82 duty-cycling normally at 1200/337/30, six nights of eight hours scored sleep. Only the capture cadence differs:

1 Hz    -> duty_cycled     fail
1/5min  -> absent          feature_absent

The evidence bar does not catch it because ABSENT_MIN_RECORDS and ABSENT_MIN_ASLEEP_SPAN_S both measure duration, and this is a phase problem. A 300 s cadence against a 1200 s period only ever occupies 4 residues mod the period, so it either always hits the window or never does — 576 records and 48 h of sleep clear both bars comfortably.

This matters more than a normal misclassification because it is asymmetric. feature_absent removes the device from eligible, so all_pass is computed without it. A device that would have blocked promotion instead silently drops out, and the bar is all_pass and ≥2 devices. It makes promoting easier, which is the wrong direction for a safety gate.

Suggested fix — the quantity is already there. record_interval_s is computed even when mode == "absent", so gate the claim on the capture being able to see a window at all:

feature_absent = (
    duty["mode"] == "absent"
    and len(records) >= ABSENT_MIN_RECORDS
    and asleep_span >= ABSENT_MIN_ASLEEP_SPAN_S
    and duty["record_interval_s"] <= NOMINAL_DUTY_WINDOW_S   # else the capture could have skipped it
)

with NOMINAL_DUTY_WINDOW_S = 30 carrying the same "reference strap, not a law" caveat the period constant already has. A coarser capture stays a plain FAIL, or gets its own inconclusive bucket if you would rather not conflate it. Worth a test planting a working strap sampled at a cadence that aliases with the period — that is the case the current tests do not have.

Everything else reads well, and the per-window aggregation argument is convincing on its own. Tools-only, no app code, no parity surface, so the risk here is confined to how much the harness can be trusted — which is exactly what finding 3 is for.

Maintainer decision: merging as-is. The feature_absent cadence gate lands as an immediate follow-up.

[ryanbr]

Merged. The cadence gate from my review is up as #868 — it also repoints the postable privacy test, which was asserting against the flat-zero fixture that has no values to leak.

Findings 1 and 2 held up under independent testing: detect_duty_cycle recovered four schedules your tests never plant (including phase > period), and refused on 2 windows, scattered firings and jitter. Nice work.