Hook-timing diagnostics, WhisperX re-alignment, and merge-doc token fixes - #114
Merged
Conversation
render-hook-intro.js reimplemented hookClipEnd()/buildHookSections() with a stale HOOK_TAIL_PAD_UNBOUNDED_SECONDS of 0.16 instead of the canonical 0.50 in remotion/lib/hookTiming.ts, under-counting the --frames range passed to remotion render and cutting rendered hooks off before their true boundary. Now imports buildHookSections directly so the two can never diverge again. Also corrects the same stale value in CLAUDE.md's constants table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pulls computeCrossCorrelation/findBestLag/validatePeak/nextPowerOfTwo out of AudioSyncer into pure functions in scripts/lib/audioCorrelation.js, so the upcoming hook-timing diagnostic script can reuse this exact math for audio drift detection instead of creating a third copy of it. AudioSyncer delegates to the shared module; existing tests pass unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds scripts/diagnostics/verify-hook-timing.ts (npm run diagnose:hooks), the first layer of a reusable diagnostic for the "hooks cut off too early/late" class of bugs. Recomputes expected hook sections directly from transcript.json via the canonical getHookSubClips()/hookClipEnd() in remotion/lib/hookTiming.ts and reports per-hook source windows, frame ranges, and durations. No media I/O — fast, always-safe first-line check that would have caught the render-hook-intro.js drift fixed earlier in this branch before it ever produced a bad render. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds --rendered/--source flags to verify-hook-timing.ts: extracts each hook's audio from a rendered output and from the original synced source video, cross-correlates them via the shared FFT utilities, and flags any hook whose measured drift exceeds --tolerance-ms. This is the black-box check that catches Remotion encode-time rounding or dropped frames that the math-consistency layer alone can't see. New scripts/lib/extractAudioWindow.js handles ffmpeg extraction + wav loading. Its wavefile import uses a namespace import (not the default- import-then-destructure pattern AudioSyncer.js uses) — that pattern throws under Jest's CJS/ESM interop for this package once wavefile isn't mocked, which AudioSyncer's own mocked tests never exercise. tests/integration/verify-hook-timing-audio.test.ts validates the whole extraction+correlation path against real ffmpeg and real (unmocked) FFT math using a deterministic seeded noise fixture, confirming ~0ms lag for a matching window and accurate measurement of a deliberately introduced 50ms offset. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds --verify-content to verify-hook-timing.ts: transcribes each hook's already-extracted rendered audio via whisper.cpp (Transcriber.js, base.en model — small/fast, this only needs a rough word list) and diffs it against the segment's expected spoken words (bounded by its own hookFrom/hookTo, not the tail-padded sourceEnd). Flags missing/extra words, catching wrong-phrase bugs — e.g. resolvePhraseToTimeRange silently falling back to the whole segment — that pure timing checks can't see. Off by default (slow, may trigger a model download). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1. verify-hook-timing.ts compared rendered audio against the *unrounded*
[sourceStart, sourceEnd) window instead of the frame-rounded
[trimBefore, trimAfter) window Remotion actually renders — a
systematic up-to-1-frame bias baked into the check itself. Running
the full pipeline against a real render (npm run render:hook-intro)
surfaced this: every one of 35 hooks showed an identical ~50-66ms
lag before the fix, and a perfectly uniform 50.0ms after — the
uniformity across unrelated hooks was the tell that this was a
comparison bug, not per-hook drift.
2. extractAudioWindow.js's wavefile import broke depending on how it's
loaded: a namespace-then-destructure worked under Jest but under tsx
executing a real .ts file (the actual production path, as opposed to
`tsx -e`) the CJS/ESM interop wraps the module as { default: {
WaveFile } } instead of exposing it directly. Neither shape alone
covers both runners; now defensively checks both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Flags any hook whose hookFrom/hookTo doesn't overlap any of its own segment's spoken token timestamps. Found by manually investigating a user report that a hook's audio didn't match its caption — the segment's Whisper/WhisperX word alignment was compressed/wrong (all four words crammed into a 0.4s span that doesn't match natural speech pacing), so the doc's explicit HOOK timestamp — set correctly by ear — didn't overlap the (bad) token positions at all. This check catches that structurally, for free, with no media I/O, at exactly the row the content-diff layer silently missed (expectedWords was empty for these rows, so an empty-vs-empty diff falsely "passed"). This is a "verify by ear" signal, not pass/fail: a correctly hand-fixed boundary still won't overlap the underlying bad tokens, so it will keep firing on rows that are already fine. Documented as such in the row output and --help text. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds diagnoseHookEnding() to the content-diff layer: compares the
transcribed rendered clip's actual last word against the hook's
intended final word (hookPhrase, or the segment's own text), and on a
mismatch, auto-triages the root cause:
- code-bug a token for the word exists inside hookClipEnd's
own search window but wasn't used to extend the
clip — a real regression to fix in hookTiming.ts.
- needs-retiming a token exists outside the window, but the
segment's alignment is otherwise trustworthy —
widen hookTo (a suggested value is computed).
- bad-alignment-data no trustworthy token exists for the word at all —
can't be auto-fixed, needs a human/agent to
verify the true ending by listening.
The zero-token-overlap check (previous commit) only caught the extreme
case where a segment's entire token set is disjoint from its hookFrom/
hookTo. A follow-up report that other hooks still end too early, plus a
manual spot-check of 5 hooks it didn't flag (4 of 5 genuinely
truncated), showed the more common failure mode is a hook whose tokens
nominally overlap the window but whose hookTo still lands before the
true spoken ending — this check catches that by verifying against the
actual rendered audio's content instead of just token structure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous commit's diagnoseHookEnding() classified an ending mismatch as a hookClipEnd() code regression whenever the matching token's t_dtw fell inside hookClipEnd()'s own search window. That reasoning is wrong by construction: hookClipEnd() always extends sourceEnd to cover whichever in-window spoken token has the LARGEST t_end, so any token found by text-matching is necessarily one of the candidates already considered — its t_end can never exceed what hookClipEnd() used. An in-window mismatch can therefore never indicate a real code bug; it can only mean the matched token's own timestamp is wrong. Confirmed directly on hook #4 of the ragtech transcript: hookClipEnd() computed sourceEnd=63.466 (called directly to verify), correctly covering the "makes" token's t_end=63.446 — yet both the real rendered clip and the raw source video, transcribed independently, say "...where every time there's a..." trailing into silence. The token data for "the/agentic/tool/makes" doesn't correspond to real speech at that position (a stumbled false start got the clean phrase's timestamp, not a code regression). Collapses 'code-bug' and 'needs-retiming' into a single 'needs-verification' diagnosis: reports the matched token's timing as a starting point to check by ear, explicitly not a confirmed fix. Renamed suggestedHookTo -> candidateHookTo to make that non-guarantee clear in the API too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixing hookFrom/hookTo doesn't fix hook captions: HookOverlay's buildCaptions() filters a segment's tokens to those whose t_dtw falls within [hookFrom, hookClipEnd()) — for hooks whose tokens are compressed entirely outside that window (the same alignment-quality issue behind the earlier hookFrom/hookTo fixes), buildCaptions() returns [] and no caption renders for the whole hook, even though audio/video play correctly. No override field exists on Segment to substitute caption timing independent of tokens. Adds scripts/align/realign-hooks.js: re-runs the same WhisperX forced- alignment primitive the pipeline already uses for the whole episode (run_whisperx_align.py), scoped to each hook's own window and text. Uses a purpose-built merge instead of reusing align-transcript.js's proportional-remap fallback for unmatched tokens — that assumes every token in a segment belongs within the new window, which is false whenever hookPhrase is a subset of the segment's words. Confirmed by hitting the bug directly: remapping unmatched tokens from their old (already-moved) position on a second run collapsed every token to the window's end. Matched tokens now get fresh WhisperX timestamps; everything else is deterministically excluded instead. Also exports applyAlignment/spawnPython from align-transcript.js for reuse, and fixes a real bug found while building this: an unmatched token's stale t_end was never cleared after its t_dtw got remapped forward, producing an invalid t_end < t_dtw state. Verified on the real transcript: the zero-token-overlap check (previously flagging 6 hooks) now reports none across all 36 hooks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
whisperx.align() can silently truncate its returned words at an internal sentence/pause boundary, even when given the full segment text and a window wide enough to cover it, with no error or any other signal — the returned segment's own text field is just a prefix of the input. Confirmed directly: a segment "...evaluate. Is that what it means?" returned words only through "evaluate.", dropping the second sentence entirely. This is a general whisperx.align() limitation, not specific to hook alignment — it would silently degrade the whole-episode alignment pipeline too, for any raw segment spanning more than one sentence. Adds align_with_retry() in run_whisperx_align.py: detects incomplete word coverage and retries aligning just the remaining text within the remaining window until fully covered or 4 passes are exhausted, logging a warning if still incomplete (previously silent either way). Verified against the exact failing case before (5/15 words, silently) and after (15/15) the fix, then re-ran the full scoped re-alignment from the previous commit and confirmed zero coverage warnings across all 33 hooks. Render-and-inspect confirmed the previously-missing caption now displays. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes two compounding bugs in mergeAlignedWords(), found by
investigating a detailed 16-item caption-accuracy report from watching
the actual hook preview:
1. Whisper's own tokens are frequently BPE sub-word split (" orchest" +
"rate", " Comp" + "ounding"), but WhisperX's alignment output is
whole-word ("orchestrate"). Comparing each raw token individually
against whole words meant neither half of a split word ever
matched, silently dropping the whole word from captions — exactly
why "orchestrate" and "Compounding" were missing. Tokens are now
grouped into words (leading-space heuristic, matching the rest of
this codebase) before matching, then a matched word's [start, end]
is distributed evenly across its constituent sub-tokens.
2. Matched words are now clamped to t_dtw >= sourceStart. Confirmed
precisely: hookFrom - firstToken.t_dtw was exactly 0.500 across six
unrelated hooks/words — an internal WhisperX context-padding
behavior, not speech variance. Unclamped, this either dropped a
word from HookOverlay's caption filter (no t_end to survive the
early-start fallback) or let it survive while an adjacent
no-leading-space continuation didn't, concatenating unrelated words
with no space ("these buzzwords" -> "thesewords"). Safe
unconditionally: a matched word is by definition part of the
aligned phrase, so it belongs inside the hook's own window.
Verified against every hook named in the user's report via direct
token inspection; all now show complete, correctly-positioned words.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of a recurring regression: every hook timing fix from scripts/align/realign-hooks.js was silently reverted to the original raw/compressed t_dtw/t_end the moment ANY doc edit triggered a merge-doc run, even for segments the edit didn't touch. The "preserve manual edits" step in edit-transcript.js's main() always rebuilt token timing from the fresh raw parse, using the matched previous-run token only to carry forward text corrections and the cut flag — never t_dtw/t_end. On top of that, the match itself was keyed by t_dtw value + occurrence count, which necessarily breaks once t_dtw is the thing being corrected. Adds mergeTokenFields(t, p, preserveTiming), matching tokens primarily by array position (stable across a realign-hooks.js run, which only changes timing, never token count/order) instead of by t_dtw value. When positions align, t_dtw/t_end now carry forward from the matched token. Falls back to the original value-based matching (timing not preserved) only when token count differs — a real re-transcription, where positional correspondence can't be assumed. All 113 existing edit-transcript tests plus 50 integration tests pass unchanged; added 6 new tests for mergeTokenFields covering timing preservation, the non-preserving fallback, text correction alongside timing, cut-flag carry-forward, punctuation non-inheritance, and clearing rather than fabricating a missing t_end. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mergeTokenFields (added in a739a78) adds timestampOffset back onto a preserved token's t_dtw *and* t_end so the later single subtraction pass lands on the intended value. But that later pass only subtracted off from t_dtw, never t_end — so t_end drifted by +timestampOffset on every merge-doc run while t_dtw stayed correct. Confirmed live: every hook's t_end grew by exactly +0.5s (this project's offset) per run. Extracted the offset-application block into applyTimestampOffset() so it's unit-testable, and made it subtract off from t_end symmetrically with t_dtw. Verified idempotence by running merge-doc three times in a row on the live transcript and diffing hook token timing between each run — zero drift after this fix (previously drifted every run). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
applyTextPartsToTokens' LCS-mismatch path (doc has more words than
token groups) correctly synthesizes new tokens for the doc's extra
words, but only ever added tokens — it never removed a token group
that no doc word maps to anymore once its replacement was
synthesized. Found via hook #302 ("...harness and loop and
compound"): whisper.cpp misheard "loop and" as "lupin" (tokenized as
" l"+"up"+"in"); the user corrected the doc text, LCS synthesized new
"loop"/"and" tokens, but the leftover "lupin" tokens stayed in the
array uncut, still spelled "lupin", rendering as spoken caption/audio
content the doc no longer asks for ("loop and compound" showed as
"loop and lupin compound"). Not specific to this hook — reproduces
for any word-count-changing correction where LCS finds zero overlap
between old and new spelling.
Marks every token in every LCS-unmatched group as cut: true. Matched
groups already get corrected text in place; unmatched ones now get
removed from output instead of lingering with their old text.
Added a lupin->loop-and regression test; updated the existing "a
lot's"->"award's" test (same code path) to assert the leftover
tokens are now cut rather than merely absent from segment.text.
Also documents the also-newly-added Commit 12/13 entries in
HOOK_TIMING_DIAGNOSTICS.md (token-timing preservation across
merge-doc runs, and the t_end offset-compounding fix), which were
implemented and committed in earlier commits but not yet written up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…zard resume fixes - Diarizer.js: assignSpeakers now prefers per-word overlap against diarization turns (_tokenLevelSpeaker) before falling back to whole-segment overlap (_segmentOverlapSpeaker). Whisper's pre-alignment segment boundaries are coarse and include leading/trailing silence, which biased the old whole-segment method at fast speaker handoffs. - extract-speaker-candidates.py: extracted _sample_timestamps() and added exclude/index-start/append support so the thumbnail wizard can fetch additional candidates for a speaker without regenerating already-seen ones. - generate-thumbnail.js: detects a Remotion render that exits 0 and writes a file but produced no visual content (composition data never loaded) via an all-channels-zero check, instead of silently accepting a blank thumbnail. Also clears any stale output file before rendering so the check can't see a leftover from a previous failed attempt. - wizard.js: guards against a stale audio file left in the input dir from a prior episode by comparing its mtime against the synced video; fixes the "optimize" step's resumeStep/redoStepId condition and syncResults reuse when resuming a run; skips the redundant speaker-name review prompt when names were just assigned in the same run; and wires the thumbnail candidate-selection prompt to extract-speaker-candidates.py's new "more" flow. - Adds export-property-spec-ai-eng-buzzwords.ts: generates the poddedit Rust compositor's Tier 1 acceptance-baseline properties.json for the "ai-eng-buzzwords" fixture episode (multi-angle, real cut checkpoints) — writes to the poddedit repo, not this one. Adapted from the existing export-property-spec.ts generator; see its header for full provenance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cherry-picked the code-only commits from
ep/loop-eng(episode artifact commit13b8f0b— transcript JSON, camera detections, thumbnails, hook-intro render — is excluded).scripts/diagnostics/verify-hook-timing.ts: math-consistency, audio cross-correlation, whisper content-diff, zero-token-overlap, and ending-completeness checks, built up incrementally with fixes found while running it against a real render.scripts/align/realign-hooks.js: scoped WhisperX re-alignment for hook captions, with fixes for BPE sub-token grouping and silent multi-sentence truncation retries.scripts/edit-transcript.js: fixes so token timing (t_end), cut token groups, and speaker attribution survive repeatedmerge-docruns.scripts/diarize/Diarizer.js: token-level speaker attribution (per-word overlap) instead of whole-segment overlap.scripts/thumbnail/*: candidate re-roll support and blank-thumbnail detection.scripts/wizard.js: resume-flow fixes (stale audio detection, resumeStep/redoStepId, skip redundant prompts).scripts/lib/audioCorrelation.js: FFT correlation utility extracted fromAudioSyncer.jsfor reuse by the new diagnostic.scripts/render-hook-intro.js: fixed hook-pad constant drift.scripts/export-property-spec-ai-eng-buzzwords.ts: acceptance-baseline properties.json generator for the poddedit Rust compositor fixture.Test plan
npx tsc --noEmitpassesnpm testpasses (409 passed, 2 skipped)git diffbetweenep/loop-engand this branch, excludingpublic/, is empty — confirms no code was dropped and no artifacts were included.gitignorecheck) passed🤖 Generated with Claude Code