Skip to content

Add line-caption generator for raw portrait shorts - #91

Merged
natashaannn merged 9 commits into
mainfrom
feature/line-captions
Aug 26, 2026
Merged

Add line-caption generator for raw portrait shorts#91
natashaannn merged 9 commits into
mainfrom
feature/line-captions

Conversation

@natashaannn

@natashaannn natashaannn commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

Adds a standalone pipeline that takes a raw short-form portrait video and produces a burned-in-caption render, independent of the existing longform/shorts camera+hook pipeline:

  • npm run captions:create -- --video <path> [--num-speakers N] — copies the source video, extracts audio, transcribes (Transcriber), force-aligns for exact word boundaries, diarizes + assigns speakers when --num-speakers > 1, chunks the transcript into fixed 3-word CaptionLines (merging whisper's BPE sub-word tokens back into whole words so a line never splits mid-word, and never spanning a segment/speaker boundary so a line never mixes two speakers), and writes public/line-captions/{id}/lines.json + a human-editable lines.doc.txt.
  • npm run captions:merge -- --id <slug> — reparses lines.doc.txt after a human reword pass and overwrites only the text field per [id]; startMs/endMs/speaker stay untouched (best-effort timing, no re-alignment).
  • npm run captions:render -- --id <slug> — renders the new LineCaptionClip-{id} Remotion composition (plain video pass-through, no jump cuts/camera profiles, plus the new LineCaptionOverlay) to public/renders/.

Reuses existing primitives unmodified: Transcriber, align-transcript.js, diarize-audio.js/assign-speakers.js, the SHORT_IDS per-clip Composition registration pattern in Root.tsx, and stampMetadata.

Also includes two unrelated pre-existing fixes discovered while getting this branch to push cleanly:

  • scripts/config/paths.test.ts hardcoded POSIX path separators, failing all 26 of its assertions on Windows. Rewritten to build expectations with path.join (test-only change; paths.ts itself was already correct).
  • .husky/pre-push's secret-token scan wasn't excluding .venv/, so a base64 font-glyph blob in a vendored PIL file false-positived as a leaked API key and blocked every push on machines with a local .venv. Added --exclude-dir=.venv alongside the existing node_modules/.next exclusions.

Full design rationale and the step-by-step build log live in docs/implementation-guides/LINE_CAPTION_SHORTS.md. New lines.json schema and source files are documented in CLAUDE.md.

Test plan

  • npm run test:unit — full suite passes (280 passed, 2 pre-existing skips, 0 failed), including new unit tests: chunkLines.test.js (BPE-word reconstruction, punctuation attach, speaker-boundary breaking, cut exclusion, t_end fallback), create-line-captions.test.js (buildLineDoc formatting), merge-line-captions.test.js (doc parsing + text-only merge), LineCaptionOverlay.test.tsx (active-line selection, speaker tinting).
  • tsc --noEmit — passes.
  • npm run lint — passes on all new/changed files.
  • Pre-push hook (full jest suite, debugger/.only()/secret scans, npm audit, runtime-dir gitignore check) — passes clean.
  • Manual, not yet run (needs a real sample clip + this repo's WhisperX/Docker alignment setup, unavailable in the sandbox this branch was built in):
    1. npm run captions:create -- --video <sample.mp4> --num-speakers 2 on a short real two-speaker portrait clip.
    2. Open public/line-captions/{id}/lines.doc.txt — confirm 3-word lines read naturally, grouped under === SPEAKER === headers, and no line mixes two speakers.
    3. Hand-edit one line's wording.
    4. npm run captions:merge -- --id {id}, confirm only that line's text changed in lines.json and its startMs/endMs are untouched.
    5. npx remotion studio, select LineCaptionClip-{id}, confirm captions are burned in, timed correctly against the audio, and the edited line shows its new wording during its original time window.
    6. npm run captions:render -- --id {id}, confirm public/renders/{id}.mp4 is produced.

🤖 Generated with Claude Code

@ytexplorer ytexplorer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manual end-to-end test on Windows — 3 render fixes

Nice work on this pipeline. I pulled the branch and ran the unchecked manual test-plan box end-to-end on Windows 11 (Node 24, Python 3.12 venv, whisper.cpp medium.en, whisperx, diarize) against a real 52 s two-speaker portrait clip.

Works great: captions:create (transcribe → align → diarize → chunk) is solid — diarization found 2 real speakers, lines.doc.txt is correctly chunked with no speaker-mixing, and captions:merge changed only the edited line's text across all 73 lines (timings/speaker preserved).

Three issues blocked captions:render; all three are fixed on my branch ytexplorer:feature/line-captions (full jest suite + lint + audit pass), and I've left inline suggestions below for the two that live in this file:

  1. --props inline JSON breaks on Windows — with spawn(shell:true), cmd.exe strips the quotes and Remotion gets unparseable JSON, aborting before bundling. → props via temp file (inline suggestion).
  2. --outName isn't honored by remotion render — combined with Config.setOutputLocation('public/renders') the file lands at public/renders.mp4 instead of public/renders/<id>.mp4. → positional output path + ensureDir (inline suggestion).
  3. Remotion version mismatch (lockfile)remotion/@remotion/gif/captions/sfx/install-whisper-cpp/eslint-plugin are pinned at 4.0.451 while @remotion/cli/bundler/renderer/media-parser resolve to 4.0.477. Remotion requires the whole family on a single version, so this split lockfile fails LineCaptionClip renders (the core remotion at 4.0.451 vs the 4.0.477 tooling is the main gap). Can't suggest inline (it's the lockfile), but it's ready to cherry-pick: ytexplorer/deckcreate@6b9da81 — realigns the whole family to 4.0.477.

Comment thread scripts/line-captions/render-line-captions.js
Comment thread scripts/line-captions/render-line-captions.js
Comment thread scripts/line-captions/render-line-captions.js Outdated
natashaannn and others added 8 commits August 26, 2026 14:50
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reconstructs whisper's BPE-level tokens into whole words before bucketing
into fixed-size caption lines, so a line never splits a word or counts
punctuation as its own slot. Lines never span a segment boundary, which
keeps two speakers from ever sharing a line.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Orchestrates the raw-portrait-video-to-caption-lines pipeline: copy source,
extract audio, transcribe, force-align for exact word boundaries, diarize
and assign speakers when --num-speakers > 1, then chunk into 3-word lines
and write lines.json + a human-editable lines.doc.txt. Reuses the existing
Transcriber class and align/diarize CLI scripts unmodified.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-parses lines.doc.txt after a human reword pass and overwrites only the
text field of each matching lines.json entry by [id] — startMs/endMs/speaker
are left untouched, so the caption keeps its original best-effort time
window regardless of how much the wording changed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A plain video pass-through (no jump cuts, no camera profiles) plus
LineCaptionOverlay, which shows fixed-window 3-word caption lines instead
of the timing-driven grouping CaptionOverlay uses. Registers one
LineCaptionClip-{id} composition per public/line-captions/*/lines.json,
mirroring the existing ShortFormClip-{id} pattern in Root.tsx.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Modeled on scripts/shorts/render-short.js — spawns remotion render against
the LineCaptionClip-{id} composition and writes to public/renders/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@natashaannn
natashaannn force-pushed the feature/line-captions branch from b15d3e3 to 293f490 Compare August 26, 2026 07:47
--outName isn't a recognized Remotion CLI flag, and inline --props JSON
gets its quotes stripped by cmd.exe on Windows. Write props to a temp
JSON file (per Remotion's own recommended workaround) and pass the
output path positionally so renders land in public/renders/<id>.mp4
instead of colliding on public/renders.mp4.

Addresses review feedback from PR #91.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@natashaannn
natashaannn merged commit ec438fa into main Aug 26, 2026
1 check passed
@natashaannn
natashaannn deleted the feature/line-captions branch August 26, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants