User story
As a video editor producing a short in Mandarin, Hindi, or Bahasa Indonesia for a broader audience, I want the burned-in captions to show both the original-language line and its English translation together, so that non-speakers of the source language can still follow along.
As a viewer who speaks the source language but not English, I want the original-language line to remain the visually primary line, so that translation is a supplement, not a replacement.
Background
The line-caption pipeline (docs/implementation-guides/LINE_CAPTION_SHORTS.md) produces one text string per CaptionLine (remotion/types/lineCaptions.ts: { id, speaker, text, startMs, endMs }), and LineCaptionOverlay renders exactly that one string per active line. There is currently no concept of a second language, no translation step anywhere in the pipeline, and no dual-line layout in the renderer.
This issue depends on two others: the source language must transcribe/align/chunk correctly first (multilingual-pipeline issue), and both scripts (source + English, which is always Latin) must render legibly (font-rendering issue only matters for the non-English side here, since English is Latin).
Acceptance criteria
Happy path
Given a completed lines.json in Mandarin Chinese (or Hindi, or Bahasa Indonesia)
When I run the new translation step
Then each CaptionLine gains an English translation string, and the human-editable doc format shows both the original and translated text per line for review/correction.
Given a lines.json with per-line translations populated
When LineCaptionClip renders
Then each active caption window shows both the source-language line and its English translation simultaneously, with the source-language line visually primary (e.g. larger/higher) and the English line secondary (e.g. smaller/below), consistent with the brand's existing typography scale.
Given an English-source video (no translation needed)
When the existing single-language pipeline/renderer runs
Then behavior is completely unchanged — bilingual rendering only activates when a translation is present.
Error path / edge case
Given a line where translation failed or was skipped
When it renders
Then it falls back to showing only the original-language line (no broken layout, no empty second line taking up space).
Given hand-edited translated text in the doc format
When captions:merge runs
Then it updates the translated text field the same way it currently updates text, without disturbing startMs/endMs/speaker.
Out of scope
- Multilingual transcription/alignment/chunking itself (tracked in the multilingual-pipeline issue — this issue assumes a correct source-language
lines.json already exists).
- Non-Latin font glyph rendering (tracked in the font-rendering issue — this issue assumes both scripts already render legibly).
- Machine-translation quality/provider evaluation beyond picking a workable option (see Implementation details) — treat translation-quality tuning as a future follow-up.
- Translating in the other direction (English source → other-language subtitles) — this issue is source-language + English only, matching the stated ask.
Technical context
remotion/types/lineCaptions.ts — CaptionLine needs a new optional field, e.g. translatedText?: string, alongside the existing { id, speaker, text, startMs, endMs }.
scripts/line-captions/create-line-captions.js — Step 5/5 (chunking, create-line-captions.js:139-151) is where a new translation step would slot in, after chunkIntoLines produces the source-language lines.
scripts/line-captions/merge-line-captions.js doc-parsing regex (/^\[(\d+)\]\s*(.*)$/ per the implementation guide, Commit 3) needs a compatible extension to also capture a translated-text line per caption id — needs a doc format design that keeps [id] markers parseable (e.g. a second [id-en] line, or a fixed two-line-per-id block).
remotion/components/LineCaptionOverlay.tsx (per implementation guide Commit 4, prop { lines: CaptionLine[]; brand: Brand }) needs a dual-line layout branch when translatedText is present on the active line.
- Translation approach: no translation API/library is currently used anywhere in this codebase (confirmed no existing translation dependency) — this issue needs to pick one (e.g. a call to an LLM API already used elsewhere in the project, or a dedicated translation API) and add whatever config/env var it requires.
Implementation details
- Add
translatedText?: string to CaptionLine in remotion/types/lineCaptions.ts, and document it in CLAUDE.md's Data Schemas section per the repo convention for new artifact fields.
- Add a translation step to
create-line-captions.js (or a new standalone script, e.g. scripts/line-captions/translate-line-captions.js, run explicitly rather than folded silently into captions:create — keep it a distinct, skippable pipeline stage) that populates translatedText per line via a chosen translation provider.
- Extend
buildLineDoc() (create-line-captions.js:65-80) and the corresponding parser in merge-line-captions.js to round-trip both the original and translated text per line in lines.doc.txt.
- Add a dual-line rendering branch to
LineCaptionOverlay.tsx: when translatedText is present, render both lines stacked with the source line visually primary; when absent, render exactly as today (single line).
- Add a new npm script (e.g.
captions:translate) and document the updated pipeline order (create → translate → merge → render) in docs/implementation-guides/LINE_CAPTION_SHORTS.md.
Additional test scenarios
- Unit test: doc round-trip (write → parse) preserves both original and translated text per line, including for lines with no translation.
- Unit test:
merge-line-captions.js leaves translatedText untouched when only the original text is hand-edited, and vice versa.
- Component smoke test:
LineCaptionOverlay renders two stacked lines when translatedText is present, and exactly one line when it's absent, at the same active frame.
- Integration test: running the translation step against a small fixture
lines.json produces non-empty translatedText for every line.
Hard constraints
- Must not change rendering or data shape for any existing single-language content —
translatedText is additive and optional.
- Must not silently drop translation failures — a failed/missing translation must be visibly distinguishable from a deliberate skip (e.g.
translatedText: undefined in both cases is fine, but the translation step must log/report failures rather than writing an empty string silently).
- New
lines.json field must be declared in remotion/types/lineCaptions.ts and documented in CLAUDE.md, per the repo's Agent Implementation Convention.
Dependency issues
- Depends on the multilingual-pipeline issue (needs a correct source-language
lines.json to translate from).
- Depends on the CJK/Devanagari font-rendering issue (needs the source-language script to render legibly alongside English).
User story
As a video editor producing a short in Mandarin, Hindi, or Bahasa Indonesia for a broader audience, I want the burned-in captions to show both the original-language line and its English translation together, so that non-speakers of the source language can still follow along.
As a viewer who speaks the source language but not English, I want the original-language line to remain the visually primary line, so that translation is a supplement, not a replacement.
Background
The line-caption pipeline (
docs/implementation-guides/LINE_CAPTION_SHORTS.md) produces onetextstring perCaptionLine(remotion/types/lineCaptions.ts:{ id, speaker, text, startMs, endMs }), andLineCaptionOverlayrenders exactly that one string per active line. There is currently no concept of a second language, no translation step anywhere in the pipeline, and no dual-line layout in the renderer.This issue depends on two others: the source language must transcribe/align/chunk correctly first (multilingual-pipeline issue), and both scripts (source + English, which is always Latin) must render legibly (font-rendering issue only matters for the non-English side here, since English is Latin).
Acceptance criteria
Happy path
Given a completed
lines.jsonin Mandarin Chinese (or Hindi, or Bahasa Indonesia)When I run the new translation step
Then each
CaptionLinegains an English translation string, and the human-editable doc format shows both the original and translated text per line for review/correction.Given a
lines.jsonwith per-line translations populatedWhen
LineCaptionCliprendersThen each active caption window shows both the source-language line and its English translation simultaneously, with the source-language line visually primary (e.g. larger/higher) and the English line secondary (e.g. smaller/below), consistent with the brand's existing typography scale.
Given an English-source video (no translation needed)
When the existing single-language pipeline/renderer runs
Then behavior is completely unchanged — bilingual rendering only activates when a translation is present.
Error path / edge case
Given a line where translation failed or was skipped
When it renders
Then it falls back to showing only the original-language line (no broken layout, no empty second line taking up space).
Given hand-edited translated text in the doc format
When
captions:mergerunsThen it updates the translated text field the same way it currently updates
text, without disturbingstartMs/endMs/speaker.Out of scope
lines.jsonalready exists).Technical context
remotion/types/lineCaptions.ts—CaptionLineneeds a new optional field, e.g.translatedText?: string, alongside the existing{ id, speaker, text, startMs, endMs }.scripts/line-captions/create-line-captions.js— Step 5/5 (chunking,create-line-captions.js:139-151) is where a new translation step would slot in, afterchunkIntoLinesproduces the source-language lines.scripts/line-captions/merge-line-captions.jsdoc-parsing regex (/^\[(\d+)\]\s*(.*)$/per the implementation guide, Commit 3) needs a compatible extension to also capture a translated-text line per caption id — needs a doc format design that keeps[id]markers parseable (e.g. a second[id-en]line, or a fixed two-line-per-id block).remotion/components/LineCaptionOverlay.tsx(per implementation guide Commit 4, prop{ lines: CaptionLine[]; brand: Brand }) needs a dual-line layout branch whentranslatedTextis present on the active line.Implementation details
translatedText?: stringtoCaptionLineinremotion/types/lineCaptions.ts, and document it inCLAUDE.md's Data Schemas section per the repo convention for new artifact fields.create-line-captions.js(or a new standalone script, e.g.scripts/line-captions/translate-line-captions.js, run explicitly rather than folded silently intocaptions:create— keep it a distinct, skippable pipeline stage) that populatestranslatedTextper line via a chosen translation provider.buildLineDoc()(create-line-captions.js:65-80) and the corresponding parser inmerge-line-captions.jsto round-trip both the original and translated text per line inlines.doc.txt.LineCaptionOverlay.tsx: whentranslatedTextis present, render both lines stacked with the source line visually primary; when absent, render exactly as today (single line).captions:translate) and document the updated pipeline order (create → translate → merge → render) indocs/implementation-guides/LINE_CAPTION_SHORTS.md.Additional test scenarios
merge-line-captions.jsleavestranslatedTextuntouched when only the original text is hand-edited, and vice versa.LineCaptionOverlayrenders two stacked lines whentranslatedTextis present, and exactly one line when it's absent, at the same active frame.lines.jsonproduces non-emptytranslatedTextfor every line.Hard constraints
translatedTextis additive and optional.translatedText: undefinedin both cases is fine, but the translation step must log/report failures rather than writing an empty string silently).lines.jsonfield must be declared inremotion/types/lineCaptions.tsand documented inCLAUDE.md, per the repo's Agent Implementation Convention.Dependency issues
lines.jsonto translate from).