Add line-caption generator for raw portrait shorts - #91
Conversation
There was a problem hiding this comment.
Manual end-to-end test on Windows — 3 render fixes
Nice work on this pipeline. I pulled the branch and ran the unchecked manual test-plan box end-to-end on Windows 11 (Node 24, Python 3.12 venv, whisper.cpp medium.en, whisperx, diarize) against a real 52 s two-speaker portrait clip.
Works great: captions:create (transcribe → align → diarize → chunk) is solid — diarization found 2 real speakers, lines.doc.txt is correctly chunked with no speaker-mixing, and captions:merge changed only the edited line's text across all 73 lines (timings/speaker preserved).
Three issues blocked captions:render; all three are fixed on my branch ytexplorer:feature/line-captions (full jest suite + lint + audit pass), and I've left inline suggestions below for the two that live in this file:
--propsinline JSON breaks on Windows — withspawn(shell:true), cmd.exe strips the quotes and Remotion gets unparseable JSON, aborting before bundling. → props via temp file (inline suggestion).--outNameisn't honored byremotion render— combined withConfig.setOutputLocation('public/renders')the file lands atpublic/renders.mp4instead ofpublic/renders/<id>.mp4. → positional output path +ensureDir(inline suggestion).- Remotion version mismatch (lockfile) —
remotion/@remotion/gif/captions/sfx/install-whisper-cpp/eslint-pluginare pinned at4.0.451while@remotion/cli/bundler/renderer/media-parserresolve to4.0.477. Remotion requires the whole family on a single version, so this split lockfile failsLineCaptionCliprenders (the coreremotionat 4.0.451 vs the 4.0.477 tooling is the main gap). Can't suggest inline (it's the lockfile), but it's ready to cherry-pick:ytexplorer/deckcreate@6b9da81— realigns the whole family to4.0.477.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reconstructs whisper's BPE-level tokens into whole words before bucketing into fixed-size caption lines, so a line never splits a word or counts punctuation as its own slot. Lines never span a segment boundary, which keeps two speakers from ever sharing a line. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Orchestrates the raw-portrait-video-to-caption-lines pipeline: copy source, extract audio, transcribe, force-align for exact word boundaries, diarize and assign speakers when --num-speakers > 1, then chunk into 3-word lines and write lines.json + a human-editable lines.doc.txt. Reuses the existing Transcriber class and align/diarize CLI scripts unmodified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-parses lines.doc.txt after a human reword pass and overwrites only the text field of each matching lines.json entry by [id] — startMs/endMs/speaker are left untouched, so the caption keeps its original best-effort time window regardless of how much the wording changed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A plain video pass-through (no jump cuts, no camera profiles) plus
LineCaptionOverlay, which shows fixed-window 3-word caption lines instead
of the timing-driven grouping CaptionOverlay uses. Registers one
LineCaptionClip-{id} composition per public/line-captions/*/lines.json,
mirroring the existing ShortFormClip-{id} pattern in Root.tsx.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Modeled on scripts/shorts/render-short.js — spawns remotion render against
the LineCaptionClip-{id} composition and writes to public/renders/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
b15d3e3 to
293f490
Compare
--outName isn't a recognized Remotion CLI flag, and inline --props JSON gets its quotes stripped by cmd.exe on Windows. Write props to a temp JSON file (per Remotion's own recommended workaround) and pass the output path positionally so renders land in public/renders/<id>.mp4 instead of colliding on public/renders.mp4. Addresses review feedback from PR #91. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Adds a standalone pipeline that takes a raw short-form portrait video and produces a burned-in-caption render, independent of the existing longform/shorts camera+hook pipeline:
npm run captions:create -- --video <path> [--num-speakers N]— copies the source video, extracts audio, transcribes (Transcriber), force-aligns for exact word boundaries, diarizes + assigns speakers when--num-speakers > 1, chunks the transcript into fixed 3-wordCaptionLines (merging whisper's BPE sub-word tokens back into whole words so a line never splits mid-word, and never spanning a segment/speaker boundary so a line never mixes two speakers), and writespublic/line-captions/{id}/lines.json+ a human-editablelines.doc.txt.npm run captions:merge -- --id <slug>— reparseslines.doc.txtafter a human reword pass and overwrites only thetextfield per[id];startMs/endMs/speakerstay untouched (best-effort timing, no re-alignment).npm run captions:render -- --id <slug>— renders the newLineCaptionClip-{id}Remotion composition (plain video pass-through, no jump cuts/camera profiles, plus the newLineCaptionOverlay) topublic/renders/.Reuses existing primitives unmodified:
Transcriber,align-transcript.js,diarize-audio.js/assign-speakers.js, theSHORT_IDSper-clipCompositionregistration pattern inRoot.tsx, andstampMetadata.Also includes two unrelated pre-existing fixes discovered while getting this branch to push cleanly:
scripts/config/paths.test.tshardcoded POSIX path separators, failing all 26 of its assertions on Windows. Rewritten to build expectations withpath.join(test-only change;paths.tsitself was already correct)..husky/pre-push's secret-token scan wasn't excluding.venv/, so a base64 font-glyph blob in a vendoredPILfile false-positived as a leaked API key and blocked every push on machines with a local.venv. Added--exclude-dir=.venvalongside the existingnode_modules/.nextexclusions.Full design rationale and the step-by-step build log live in
docs/implementation-guides/LINE_CAPTION_SHORTS.md. Newlines.jsonschema and source files are documented inCLAUDE.md.Test plan
npm run test:unit— full suite passes (280 passed, 2 pre-existing skips, 0 failed), including new unit tests:chunkLines.test.js(BPE-word reconstruction, punctuation attach, speaker-boundary breaking, cut exclusion,t_endfallback),create-line-captions.test.js(buildLineDocformatting),merge-line-captions.test.js(doc parsing + text-only merge),LineCaptionOverlay.test.tsx(active-line selection, speaker tinting).tsc --noEmit— passes.npm run lint— passes on all new/changed files..only()/secret scans, npm audit, runtime-dir gitignore check) — passes clean.npm run captions:create -- --video <sample.mp4> --num-speakers 2on a short real two-speaker portrait clip.public/line-captions/{id}/lines.doc.txt— confirm 3-word lines read naturally, grouped under=== SPEAKER ===headers, and no line mixes two speakers.npm run captions:merge -- --id {id}, confirm only that line'stextchanged inlines.jsonand itsstartMs/endMsare untouched.npx remotion studio, selectLineCaptionClip-{id}, confirm captions are burned in, timed correctly against the audio, and the edited line shows its new wording during its original time window.npm run captions:render -- --id {id}, confirmpublic/renders/{id}.mp4is produced.🤖 Generated with Claude Code