feat: upgrade the captain bridge with photos and hold-to-speak

[Amplify-Logic]

Intent

Upgrade the captain's iPhone-facing Starship bridge page (bin/fm-bridge-view.py) to v1.5 with three requested improvements: multi-photo upload, hold-to-speak, and a more readable captain-first glance.

Multi-photo: the photo picker must accept multiple images in one picker session (multiple on the file input). Sequential posts per image are fine. Every image must still land as an atomic metadata+image pair in the bridge inbox exactly as single uploads do (same naming and write-then-rename discipline). Show simple progress such as "3 of 5 sent". One photo failing must be reported, not silently dropped, and must not abort the rest.

Hold-to-speak: add a press-and-hold microphone button so the captain can talk to firstmate from the bridge page as a second voice channel besides the VoiceLoop app. MediaRecorder while held; on release POST the audio to the bridge server, which forwards it into the local glasses mailbox as a new pending audio request (the same request pipeline VoiceLoop uses). Mark source "bridge" so it is distinguishable; the mailbox HTTP API and its relay token are the integration seam. The relay token stays SERVER-SIDE in the bridge process and must never ship to the browser. All new endpoints sit behind the existing passcode session. iOS Safari records audio/mp4 (m4a), not webm - whatever Safari actually produces must survive the pipeline (the Mac-side transcriber already handles m4a from VoiceLoop). After sending, show "sent" and poll for the answer; when answer text arrives, render it on the page. Text only in v1.5; no audio playback.

Page readability: restructure for a phone screen in plain outcome language. Top: anything waiting on the captain (decisions, reviews), then active work, then a COLLAPSED waiting section that expands to the full list on tap (no arbitrary truncation). Item text short and human; never raw ids/paths/repro strings as the primary line. Fast and dependency-free (no external CDNs; the page is tailnet-only).

Constraints that must not be regressed: iPhone Safari is the only client that matters. It omits/nulls Origin on same-origin POSTs (fixed in PRs #86/#87). Every new endpoint must accept client-shaped requests (Origin: null, Safari-shaped multipart/audio). Do not restart, reinstall, or point any launchd service at this worktree; the live bridge stays anchored to the permanent home. Do not change the mailbox service or its database schema beyond additive request fields. Login, single photo, and fleet view must still work. Colocate tests covering the new upload and audio endpoints (auth required, atomic writes, multi-file, Origin: null). Distinguishable bridge-origin audio may use the existing mailbox filename field (bridge.m4a) rather than a new schema field, because VoiceQuestion rejects unknown keys.

What Changed

• Restructure the phone bridge into a captain-first glance with human-readable needs, active work, and a collapsed but complete waiting list.
• Add multi-photo selection with sequential uploads, accurate progress, per-photo failure reporting, and atomic inbox metadata/image pairs.
• Add authenticated hold-to-speak audio forwarding to the glasses mailbox, server-side relay-token handling, and text answer polling, with updated documentation and endpoint coverage.

Risk Assessment

✅ Low: The latest commit correctly restores the required captain-first ordering, and the full base-to-HEAD review found no remaining material defects or intent conflicts.

Testing

The complete bridge integration suite passed, covering login and existing behavior, authenticated Origin-null multi-photo uploads with atomic pairs and partial-failure continuation, Safari-shaped m4a relay and text-answer polling, token secrecy, and browser interaction state; manual 390×844 rendering confirmed the captain-first layout and expandable full waiting list. The screenshot command did not persist its output, so reviewer-visible evidence is supplied as interactive rendered HTML instead.

Evidence: Interactive 390px Starship v1.5 phone UI evidence

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="referrer" content="no-referrer">
<title>STARSHIP</title>
<style nonce="evidence">
:root { color-scheme: dark; }
* { box-sizing: border-box; }
html, body { margin: 0; padding: 0; }
body {
  font-family: ui-sans-serif, system-ui, -apple-system, sans-serif;
  background: #101418;
  color: #f2f4f3;
  line-height: 1.45;
  padding: max(1rem, env(safe-area-inset-top)) 1.1rem 2rem;
}
main { max-width: 40rem; margin: 0 auto; }
h1 { font-size: 0.8rem; letter-spacing: 0.18em; text-transform: uppercase; margin: 0 0 0.4rem; color: #9aa7a0; }
h2 { font-size: 0.78rem; letter-spacing: 0.12em; text-transform: uppercase; margin: 1.4rem 0 0.5rem; color: #c5d0c8; }
.meta { color: #c5d0c8; font-size: 0.95rem; }
.warn { color: #e6c07b; font-size: 0.92rem; margin: 0.4rem 0 0; }
ul { list-style: none; padding: 0; margin: 0; }
li { padding: 0.55rem 0; border-bottom: 1px solid #2a3330; font-size: 1.05rem; }
.dot { display: inline-block; width: 0.65rem; height: 0.65rem; border-radius: 50%; margin-right: 0.55rem; background: #6ea8fe; }
.dot.needs { background: #c084fc; }
.dot.under { background: #6ea8fe; }
.dot.wait { background: #fbbf24; }
.dot.ready { background: #34d399; }
.dot.failed { background: #f87171; }
.dot.stuck { background: #fb923c; }
.more, .empty, .incomplete { color: #9aa7a0; font-size: 0.92rem; margin: 0.4rem 0 0; }
a { color: #9cdcfe; }
input, button {
  font: inherit; width: 100%; min-height: 2.75rem; border-radius: 0.5rem;
  border: 1px solid #3b4742; padding: 0.6rem 0.8rem;
}
input { background: #1b2220; color: inherit; margin: 0.8rem 0; }
button { background: #d7e0d8; color: #101418; font-weight: 600; }
.note { color: #9aa7a0; font-size: 0.9rem; }
.ok { color: #34d399; font-size: 0.95rem; margin: 0.4rem 0 0; }
#photo-form, #speak-box { margin: 1.1rem 0 0.4rem; }
#photo-form label, #speak-box label { display: block; margin-bottom: 0.35rem; }
#hold-speak {
  touch-action: none;
  user-select: none;
  -webkit-user-select: none;
  background: #6ea8fe;
  color: #101418;
}
#hold-speak.held { background: #f87171; color: #101418; }
#speak-answer {
  white-space: pre-wrap;
  margin: 0.6rem 0 0;
  font-size: 1.05rem;
}
details.waiting-fold {
  margin: 0.2rem 0 0;
  border: 1px solid #2a3330;
  border-radius: 0.5rem;
  padding: 0.35rem 0.7rem 0.6rem;
}
details.waiting-fold > summary {
  min-height: 2.75rem;
  display: flex;
  align-items: center;
  cursor: pointer;
  color: #c5d0c8;
  font-size: 0.95rem;
}
#stale {
  display: none; position: fixed; inset: 0; background: #101418;
  color: #f2f4f3; align-items: center; justify-content: center;
  text-align: center; padding: 2rem; font-size: 1.4rem; z-index: 9;
}
#stale.on { display: flex; }
header { display: flex; justify-content: space-between; align-items: baseline; gap: 1rem; }
form.logout { margin: 0; width: auto; }
form.logout button { width: auto; min-height: 2rem; padding: 0.3rem 0.7rem; background: transparent; color: #c5d0c8; border-color: #3b4742; }
</style>
<script nonce="evidence">
const STALE_MS = 90 * 1000;
const REFRESH_MS = 30 * 1000;
let lastSuccess = 0;
let speakBusy = false;
let speakRecorder = null;
let speakChunks = [];
let speakStream = null;
let speakPollTimer = 0;
let speakHeld = false;
let speakStarting = false;
let speakPress = 0;
function esc(value) {
  return String(value).replace(/[&<>"']/g, function(ch) {
    return ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[ch]);
  });
}
function setStale(on) {
  const el = document.getElementById('stale');
  if (!el) return;
  el.classList.toggle('on', on);
}
function setPhotoCount(n) {
  const count = document.getElementById('photo-count');
  if (count) count.textContent = 'Photos received today: ' + n;
}
function markBucketsUnreachable() {
  ['needs','underway','finished','waiting'].forEach(function(id) {
    const root = document.getElementById(id);
    if (!root) return;
    if (root.textContent.indexOf('Loading') !== -1) {
      root.innerHTML = '<p class="empty">Cannot reach the desk.</p>';
    }
  });
}
function dotClass(name) {
  if (name === 'Needs you') return 'needs';
  if (name === 'Under way') return 'under';
  if (name === 'Waiting') return 'wait';
  if (name === 'Ready') return 'ready';
  if (name === 'Failed') return 'failed';
  if (name === 'Stuck') return 'stuck';
  return 'under';
}
function renderBucket(id, bucket, emptyText) {
  const root = document.getElementById(id);
  if (!root || !bucket) return;
  const items = bucket.items || [];
  const lines = items.map(function(item) {
    const title = esc(item.title || '');
    const url = item.url || '';
    const safeUrl = /^https:\/\/github\.com\/[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+\/pull\/[0-9]+$/.test(url) ? url : '';
    const label = safeUrl
      ? '<a href="' + safeUrl + '" rel="noreferrer">' + title + '</a>'
      : title;
    return '<li><span class="dot ' + dotClass(item.dot) + '"></span>' + label + '</li>';
  });
  let extra = '';
  if (!items.length) extra = '<p class="empty">' + emptyText + '</p>';
  root.innerHTML = (lines.length ? '<ul>' + lines.join('') + '</ul>' : '') + extra;
}
function renderWaiting(bucket) {
  renderBucket('waiting', bucket, 'Nothing is waiting.');
  const summary = document.getElementById('waiting-summary');
  if (!summary) return;
  const n = (bucket && bucket.items) ? bucket.items.length : 0;
  summary.textContent = n ? ('Waiting · ' + n) : 'Waiting';
}
function apply(data) {
  lastSuccess = Date.now();
  setStale(false);
  const age = document.getElementById('observed');
  if (age) age.textContent = 'Observed just now';
  const desk = document.getElementById('desk');
  if (desk) desk.textContent = 'Desk reachable';
  const mail = document.getElementById('mailbox');
  if (mail) mail.textContent = data.mailbox_listener ? 'Mailbox on' : 'Mailbox listener off';
  renderBucket('needs', data.needs_you, 'Nothing needs you right now.');
  renderBucket('underway', data.under_way, 'Nothing is under way.');
  renderBucket('finished', data.just_finished, 'No recent completions.');
  renderWaiting(data.waiting);
  setPhotoCount(data.photos_today != null ? data.photos_today : 0);
}
function setSpeakStatus(kind, text) {
  const status = document.getElementById('speak-result');
  if (!status) return;
  status.className = kind;
  status.textContent = text;
}
function stopTracks() {
  if (!speakStream) return;
  speakStream.getTracks().forEach(function(track) { track.stop(); });
  speakStream = null;
}
function pickRecorderType() {
  if (typeof MediaRecorder === 'undefined') return '';
  const types = ['audio/mp4', 'audio/aac', 'audio/webm;codecs=opus', 'audio/webm'];
  for (let i = 0; i < types.length; i++) {
    if (MediaRecorder.isTypeSupported(types[i])) return types[i];
  }
  return '';
}
async function startSpeak(ev) {
  if (ev) ev.preventDefault();
  if (speakBusy || speakRecorder || speakStarting) return;
  speakHeld = true;
  const press = ++speakPress;
  const btn = document.getElementById('hold-speak');
  if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
    setSpeakStatus('warn', 'This phone cannot record audio here.');
    return;
  }
  speakStarting = true;
  try {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    if (!speakHeld || press !== speakPress) {
      stream.getTracks().forEach(function(track) { track.stop(); });
      return;
    }
    speakStream = stream;
    speakChunks = [];
    const mime = pickRecorderType();
    speakRecorder = mime ? new MediaRecorder(speakStream, { mimeType: mime }) : new MediaRecorder(speakStream);
    speakRecorder.addEventListener('dataavailable', function(event) {
      if (event.data && event.data.size) speakChunks.push(event.data);
    });
    speakRecorder.start();
    if (btn) {
      btn.classList.add('held');
      btn.textContent = 'Release to send';
    }
    setSpeakStatus('note', 'Listening…');
  } catch (err) {
    stopTracks();
    speakRecorder = null;
    setSpeakStatus('warn', 'Microphone was not available.');
  } finally {
    speakStarting = false;

... [656 bytes truncated] ...

move('held');
    btn.textContent = 'Hold to speak';
  }
  if (!blob.size) {
    speakBusy = false;
    setSpeakStatus('warn', 'Nothing was recorded.');
    return;
  }
  setSpeakStatus('note', 'Sending…');
  try {
    const body = new FormData();
    const name = blob.type.indexOf('webm') !== -1 ? 'bridge.webm' : 'bridge.m4a';
    body.append('audio', blob, name);
    const res = await fetch('/speak', {
      method: 'POST',
      body: body,
      credentials: 'same-origin',
      cache: 'no-store'
    });
    const payload = await res.json().catch(function() { return {}; });
    if (!res.ok) {
      speakBusy = false;
      setSpeakStatus('warn', payload.error || ('Send failed (' + res.status + ')'));
      return;
    }
    setSpeakStatus('ok', 'Sent');
    if (payload.request_id) pollAnswer(payload.request_id, 0);
    else speakBusy = false;
  } catch (err) {
    speakBusy = false;
    setSpeakStatus('warn', 'Send failed.');
  }
}
async function pollAnswer(requestId, attempt) {
  if (speakPollTimer) clearTimeout(speakPollTimer);
  try {
    const res = await fetch('/api/answer/' + encodeURIComponent(requestId), {
      credentials: 'same-origin',
      cache: 'no-store'
    });
    const payload = await res.json().catch(function() { return {}; });
    if (res.ok && payload && payload.text) {
      const box = document.getElementById('speak-answer');
      if (box) box.textContent = payload.text;
      setSpeakStatus('ok', 'Answered');
      speakBusy = false;
      return;
    }
    if (res.ok && payload && payload.pending && attempt < 150) {
      speakPollTimer = setTimeout(function() { pollAnswer(requestId, attempt + 1); }, 2000);
      return;
    }
    if (attempt < 150 && (res.status === 204 || (payload && payload.pending))) {
      speakPollTimer = setTimeout(function() { pollAnswer(requestId, attempt + 1); }, 2000);
      return;
    }
    speakBusy = false;
    if (!document.getElementById('speak-answer').textContent) {
      setSpeakStatus('warn', 'No answer yet.');
    }
  } catch (err) {
    if (attempt < 150) {
      speakPollTimer = setTimeout(function() { pollAnswer(requestId, attempt + 1); }, 2000);
      return;
    }
    speakBusy = false;
    setSpeakStatus('warn', 'Could not read the answer.');
  }
}
async function sendPhotos(ev) {
  ev.preventDefault();
  const input = document.getElementById('photo');
  const status = document.getElementById('photo-result');
  if (!input || !status) return;
  const files = input.files ? Array.prototype.slice.call(input.files) : [];
  if (!files.length) {
    status.className = 'warn';
    status.textContent = 'Choose a photo first.';
    return;
  }
  const failed = [];
  let sent = 0;
  for (let i = 0; i < files.length; i++) {
    status.className = 'note';
    status.textContent = 'Sending ' + (i + 1) + ' of ' + files.length + '…';
    try {
      const body = new FormData();
      body.append('photo', files[i]);
      const res = await fetch('/upload', {
        method: 'POST',
        body: body,
        credentials: 'same-origin',
        cache: 'no-store'
      });
      const payload = await res.json().catch(function() { return {}; });
      if (!res.ok) {
        failed.push('Photo ' + (i + 1) + ': ' + (payload.error || ('failed (' + res.status + ')')));
        continue;
      }
      sent += 1;
      status.textContent = sent + ' of ' + files.length + ' sent';
      if (payload.received_today != null) setPhotoCount(payload.received_today);
    } catch (err) {
      failed.push('Photo ' + (i + 1) + ': send failed');
    }
  }
  input.value = '';
  if (failed.length && failed.length === files.length) {
    status.className = 'warn';
    status.textContent = failed.join(' ');
  } else if (failed.length) {
    status.className = 'warn';
    status.textContent = sent + ' of ' + files.length + ' sent. ' + failed.join(' ');
  } else {
    status.className = 'ok';
    status.textContent = files.length === 1 ? 'Photo received.' : (files.length + ' of ' + files.length + ' sent');
  }
}
function tickObserved() {
  const age = document.getElementById('observed');
  if (!age) return;
  if (!lastSuccess) return;
  const seconds = Math.max(0, Math.round((Date.now() - lastSuccess) / 1000));
  age.textContent = 'Observed ' + seconds + ' seconds ago';
  if (Date.now() - lastSuccess > STALE_MS) setStale(true);
}
async function refresh() {
  try {
    const ctl = new AbortController();
    const timer = setTimeout(function() { ctl.abort(); }, 10000);
    const res = await fetch('/api/observation', { credentials: 'same-origin', cache: 'no-store', signal: ctl.signal });
    clearTimeout(timer);
    const payload = await res.json().catch(function() { return {}; });
    if (payload && payload.photos_today != null) setPhotoCount(payload.photos_today);
    if (!res.ok) throw new Error('status ' + res.status);
    apply(payload);
  } catch (err) {
    if (!lastSuccess) {
      setStale(true);
      markBucketsUnreachable();
    } else {
      tickObserved();
    }
  }
}
document.addEventListener('DOMContentLoaded', function() {
  refresh();
  setInterval(refresh, REFRESH_MS);
  setInterval(tickObserved, 1000);
  document.addEventListener('visibilitychange', function() { if (!document.hidden) refresh(); });
  window.addEventListener('pageshow', function() { refresh(); });
  const form = document.getElementById('photo-form');
  if (form) form.addEventListener('submit', sendPhotos);
  const hold = document.getElementById('hold-speak');
  if (hold) {
    hold.addEventListener('pointerdown', function(ev) {
      try { hold.setPointerCapture(ev.pointerId); } catch (err) {}
      startSpeak(ev);
    });
    hold.addEventListener('pointerup', finishSpeak);
    hold.addEventListener('pointercancel', finishSpeak);
    hold.addEventListener('lostpointercapture', finishSpeak);
  }
});
</script>
</head>
<body>
<div id="stale">Cannot reach the desk.</div>
<main>
<header>
  <h1>Starship</h1>
  <form class="logout" method="post" action="/logout"><button type="submit">Log out</button></form>
</header>
<p class="meta"><span id="desk">Checking the desk</span> · <span id="mailbox">Mailbox…</span></p>
<p class="meta" id="observed">Observing…</p>
<p class="warn">Summary only. Do not approve from this page.</p>
<h2>Needs you</h2>
<div id="needs"><p class="empty">Loading…</p></div>
<h2>Under way</h2>
<div id="underway"><p class="empty">Loading…</p></div>
<details class="waiting-fold">
  <summary id="waiting-summary">Waiting</summary>
  <div id="waiting"><p class="empty">Loading…</p></div>
</details>
<h2>Talk</h2>
<div id="speak-box">
  <label for="hold-speak">Hold to speak</label>
  <button type="button" id="hold-speak">Hold to speak</button>
  <p id="speak-result" class="note"></p>
  <p id="speak-answer" class="meta"></p>
</div>
<h2>Send photos</h2>
<form id="photo-form" method="post" action="/upload" enctype="multipart/form-data">
  <label for="photo">Photos</label>
  <input id="photo" name="photo" type="file" accept="image/jpeg,image/png,image/webp,image/heic,image/heif,image/*" multiple>
  <button type="submit">Send photos</button>
</form>
<p class="meta" id="photo-count">Photos received today: 0</p>
<p id="photo-result" class="note"></p>
<h2>Just finished</h2>
<div id="finished"><p class="empty">Loading…</p></div>
</main>
<script nonce="evidence">document.addEventListener('DOMContentLoaded',()=>{apply({"mailbox_listener": true, "photos_today": 3, "needs_you": {"items": [{"title": "Choose the release window", "dot": "Needs you"}, {"title": "Review the glasses voice demo", "dot": "Needs you"}]}, "under_way": {"items": [{"title": "Making bridge voice replies faster", "dot": "Under way"}, {"title": "Checking photo delivery on iPhone", "dot": "Under way"}]}, "waiting": {"items": [{"title": "Voice answer transcription", "dot": "Waiting"}, {"title": "Photo follow-up review", "dot": "Waiting"}, {"title": "Tailnet connection check", "dot": "Waiting"}, {"title": "Captain approval for rollout", "dot": "Waiting"}]}, "just_finished": {"items": [{"title": "Multi-photo uploads are ready", "dot": "Ready"}]}});document.getElementById('speak-result').textContent='Ready when you are.';});</script></body>
</html>

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 3 issues found → auto-fixed (2) ✅

• 🚨 bin/fm-bridge-view.py:1098 - Required criterion: “a COLLAPSED waiting section that expands to the full list on tap (no arbitrary truncation).” The diff sets FM_BEARINGS_GATES to 1000 instead of invoking the snapshot’s existing --all-queued option, so more than 1000 queued items are silently omitted while the response reports more: 0. Use --all-queued so the expanded section is genuinely complete.
• 🚨 bin/fm-bridge-view.py:1360 - Required criterion: “MediaRecorder while held; on release POST the audio.” startSpeak awaits getUserMedia before assigning speakRecorder, while finishSpeak immediately returns when no recorder exists. If the captain releases during the permission prompt or before the async request resolves, recording begins after release and continues indefinitely because that release was lost. Track the held/pointer state across the await and cancel or immediately stop when release occurred.
• ⚠️ bin/fm-bridge-view.py:1478 - Required criterion: “Show simple progress such as ‘3 of 5 sent’.” The diff displays (i + 1) of N sent before that photo’s request begins, so it can claim a photo was sent while still uploading or before it fails. Update progress after each successful response, using a neutral “Sending X of N” message while the request is pending.

🔧 Fix: Fix complete waiting, speech release, and upload progress
1 error still open:

• 🚨 bin/fm-bridge-view.py:1633 - Required criterion: “Top: anything waiting on the captain (decisions, reviews), then active work, then a COLLAPSED waiting section.” The changed markup inserts the Talk and Send photos controls between Needs you and Under way, so active work is pushed below both input panels rather than following the captain-needed section. Move the controls after the three glance sections, or confirm that this ordering is intentionally being relaxed.

🔧 Fix: Order captain glance before communication controls
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• bash tests/fm-bridge-view.test.sh
• Rendered bridge-v1.5-phone.html at a 390×844 iPhone-sized viewport using the actual bridge HTML, CSS, and JavaScript.
• Expanded the collapsed “Waiting · 4” section and verified all four human-readable items appeared without truncation.
• Visually verified captain-first ordering, hold-to-speak controls, multi-photo picker, photo count, and dependency-free phone layout.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.