Skip to content

feat(line-captions): add bilingual (English + source language) subtitle rendering #117

Description

@natashaannn

User story

As a video editor producing a short in Mandarin, Hindi, or Bahasa Indonesia for a broader audience, I want the burned-in captions to show both the original-language line and its English translation together, so that non-speakers of the source language can still follow along.

As a viewer who speaks the source language but not English, I want the original-language line to remain the visually primary line, so that translation is a supplement, not a replacement.

Background

The line-caption pipeline (docs/implementation-guides/LINE_CAPTION_SHORTS.md) produces one text string per CaptionLine (remotion/types/lineCaptions.ts: { id, speaker, text, startMs, endMs }), and LineCaptionOverlay renders exactly that one string per active line. There is currently no concept of a second language, no translation step anywhere in the pipeline, and no dual-line layout in the renderer.

This issue depends on two others: the source language must transcribe/align/chunk correctly first (multilingual-pipeline issue), and both scripts (source + English, which is always Latin) must render legibly (font-rendering issue only matters for the non-English side here, since English is Latin).

Acceptance criteria

Happy path

Given a completed lines.json in Mandarin Chinese (or Hindi, or Bahasa Indonesia)
When I run the new translation step
Then each CaptionLine gains an English translation string, and the human-editable doc format shows both the original and translated text per line for review/correction.

Given a lines.json with per-line translations populated
When LineCaptionClip renders
Then each active caption window shows both the source-language line and its English translation simultaneously, with the source-language line visually primary (e.g. larger/higher) and the English line secondary (e.g. smaller/below), consistent with the brand's existing typography scale.

Given an English-source video (no translation needed)
When the existing single-language pipeline/renderer runs
Then behavior is completely unchanged — bilingual rendering only activates when a translation is present.

Error path / edge case

Given a line where translation failed or was skipped
When it renders
Then it falls back to showing only the original-language line (no broken layout, no empty second line taking up space).

Given hand-edited translated text in the doc format
When captions:merge runs
Then it updates the translated text field the same way it currently updates text, without disturbing startMs/endMs/speaker.

Out of scope

  • Multilingual transcription/alignment/chunking itself (tracked in the multilingual-pipeline issue — this issue assumes a correct source-language lines.json already exists).
  • Non-Latin font glyph rendering (tracked in the font-rendering issue — this issue assumes both scripts already render legibly).
  • Machine-translation quality/provider evaluation beyond picking a workable option (see Implementation details) — treat translation-quality tuning as a future follow-up.
  • Translating in the other direction (English source → other-language subtitles) — this issue is source-language + English only, matching the stated ask.

Technical context

  • remotion/types/lineCaptions.tsCaptionLine needs a new optional field, e.g. translatedText?: string, alongside the existing { id, speaker, text, startMs, endMs }.
  • scripts/line-captions/create-line-captions.js — Step 5/5 (chunking, create-line-captions.js:139-151) is where a new translation step would slot in, after chunkIntoLines produces the source-language lines.
  • scripts/line-captions/merge-line-captions.js doc-parsing regex (/^\[(\d+)\]\s*(.*)$/ per the implementation guide, Commit 3) needs a compatible extension to also capture a translated-text line per caption id — needs a doc format design that keeps [id] markers parseable (e.g. a second [id-en] line, or a fixed two-line-per-id block).
  • remotion/components/LineCaptionOverlay.tsx (per implementation guide Commit 4, prop { lines: CaptionLine[]; brand: Brand }) needs a dual-line layout branch when translatedText is present on the active line.
  • Translation approach: no translation API/library is currently used anywhere in this codebase (confirmed no existing translation dependency) — this issue needs to pick one (e.g. a call to an LLM API already used elsewhere in the project, or a dedicated translation API) and add whatever config/env var it requires.

Implementation details

  1. Add translatedText?: string to CaptionLine in remotion/types/lineCaptions.ts, and document it in CLAUDE.md's Data Schemas section per the repo convention for new artifact fields.
  2. Add a translation step to create-line-captions.js (or a new standalone script, e.g. scripts/line-captions/translate-line-captions.js, run explicitly rather than folded silently into captions:create — keep it a distinct, skippable pipeline stage) that populates translatedText per line via a chosen translation provider.
  3. Extend buildLineDoc() (create-line-captions.js:65-80) and the corresponding parser in merge-line-captions.js to round-trip both the original and translated text per line in lines.doc.txt.
  4. Add a dual-line rendering branch to LineCaptionOverlay.tsx: when translatedText is present, render both lines stacked with the source line visually primary; when absent, render exactly as today (single line).
  5. Add a new npm script (e.g. captions:translate) and document the updated pipeline order (create → translate → merge → render) in docs/implementation-guides/LINE_CAPTION_SHORTS.md.

Additional test scenarios

  • Unit test: doc round-trip (write → parse) preserves both original and translated text per line, including for lines with no translation.
  • Unit test: merge-line-captions.js leaves translatedText untouched when only the original text is hand-edited, and vice versa.
  • Component smoke test: LineCaptionOverlay renders two stacked lines when translatedText is present, and exactly one line when it's absent, at the same active frame.
  • Integration test: running the translation step against a small fixture lines.json produces non-empty translatedText for every line.

Hard constraints

  • Must not change rendering or data shape for any existing single-language content — translatedText is additive and optional.
  • Must not silently drop translation failures — a failed/missing translation must be visibly distinguishable from a deliberate skip (e.g. translatedText: undefined in both cases is fine, but the translation step must log/report failures rather than writing an empty string silently).
  • New lines.json field must be declared in remotion/types/lineCaptions.ts and documented in CLAUDE.md, per the repo's Agent Implementation Convention.

Dependency issues

  • Depends on the multilingual-pipeline issue (needs a correct source-language lines.json to translate from).
  • Depends on the CJK/Devanagari font-rendering issue (needs the source-language script to render legibly alongside English).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions