Skip to content

feat(remotion): add CJK and Devanagari font support for burned-in captions #116

Description

@natashaannn

User story

As a video editor rendering a line-caption short in Mandarin Chinese or Hindi, I want the burned-in captions to display readable glyphs in the source script, so that viewers see actual Chinese/Devanagari text instead of tofu boxes or missing characters.

Background

All caption rendering in this codebase loads exactly one font family: Nunito, a Latin-script-only Google variable font, loaded unconditionally in remotion/loadFonts.ts (loadNunito()) and referenced via brand.typography.fontFamily (remotion/types/brand.ts:16) in every text-rendering overlay, including CaptionOverlay.tsx:350 (fontFamily: typography.fontFamily) and the planned LineCaptionOverlay.tsx (per docs/implementation-guides/LINE_CAPTION_SHORTS.md Commit 4, which explicitly reuses CaptionOverlay.tsx's typography constants for visual consistency).

Nunito has no glyph coverage for CJK (Mandarin Chinese) or Devanagari (Hindi) scripts. Even once transcription/alignment/chunking correctly produce Chinese or Hindi text (tracked in the multilingual-pipeline issue), rendering that text through Remotion's OffthreadVideo/browser-based compositor with only Nunito loaded will fall back to the system default font or render nothing/placeholder glyphs — the burned-in video output would be illegible for these two languages. Bahasa Indonesia uses the Latin script, so it is unaffected and out of scope here.

Acceptance criteria

Happy path

Given a lines.json containing Mandarin Chinese caption text
When LineCaptionClip/LineCaptionOverlay renders a frame with an active line
Then the Chinese text renders with correct, legible CJK glyphs in the output video (verified by a rendered frame screenshot, not just absence of a crash).

Given a lines.json containing Hindi (Devanagari) caption text
When the same composition renders
Then the Devanagari text renders with correct glyphs, including proper conjunct/ligature shaping.

Given existing English content and brand configs
When any composition renders as before
Then Nunito continues to be used for Latin-script text — this change must not alter existing English/Latin rendering.

Error path / edge case

Given caption text mixing Latin and CJK/Devanagari characters within the same line
When it renders
Then each character renders with an appropriate glyph from whichever loaded font covers it (font-family fallback list), with no visible tofu boxes.

Out of scope

  • Bilingual (dual-language, e.g. English + Chinese simultaneously) subtitle layout — tracked in a separate issue.
  • Transcription/alignment/chunking correctness for these languages — tracked in the multilingual-pipeline issue.
  • Bahasa Indonesia — Latin script, already covered by Nunito.
  • Non-caption UI text (e.g. the Next.js app's own UI chrome) — this issue is scoped to Remotion-rendered burned-in captions only.

Technical context

  • remotion/loadFonts.ts — currently exports only loadNunito(); needs additional FontFace registrations for a CJK-covering font and a Devanagari-covering font (e.g. Noto Sans SC and Noto Sans Devanagari, both free/OFL-licensed and available as static files, consistent with the existing self-hosted .ttf pattern under public/fonts/).
  • remotion/types/brand.ts:16typography.fontFamily: string is a single font name; will likely need to become a font-family fallback stack (e.g. "Nunito, 'Noto Sans SC', 'Noto Sans Devanagari', sans-serif") or a per-script lookup.
  • remotion/components/CaptionOverlay.tsx:350 and the planned LineCaptionOverlay.tsx both consume typography.fontFamily directly — a fallback-stack string change should require no changes to these consumers, but must be verified.
  • Font files must be added under public/fonts/ following the existing Nunito file naming/location convention.
  • Bundle-size impact: Noto Sans SC in particular is large (full CJK coverage); consider a subsetted build or a CDN-free self-hosted subset limited to common characters if render performance/bundle size becomes an issue — flag this as a follow-up if so, don't block this issue on it.

Implementation details

  1. Source appropriately licensed static font files covering CJK (e.g. Noto Sans SC) and Devanagari (e.g. Noto Sans Devanagari) and add them under public/fonts/.
  2. Extend remotion/loadFonts.ts with loadNotoSansSC()/loadNotoSansDevanagari() (or a combined loadCaptionFonts()), following the existing FontFace + document.fonts.add() pattern in loadNunito().
  3. Wire the new loaders into whichever delayRender/continueRender call site currently invokes loadNunito() (locate via the composition root/Composition.tsx) so fonts are loaded before render, not just for English brand.
  4. Update brand.typography.fontFamily (or add a new field) to express a font-family fallback stack so a browser/Remotion text run picks the right glyphs per character without per-language branching in overlay components.
  5. Verify CaptionOverlay.tsx and LineCaptionOverlay.tsx (once it exists) render correctly with mixed-script text using the fallback stack.

Additional test scenarios

  • Smoke render test: LineCaptionOverlay (or CaptionOverlay) with Chinese-only, Hindi-only, and mixed Latin+CJK sample text, asserting the DOM text content is present (a full pixel-level glyph-rendering assertion isn't practical in jsdom — pair this with a manual visual check via remotion studio per the acceptance criteria).
  • Manual visual test: render one short clip per target language via npx remotion studio and visually confirm legible glyphs in a captured frame.

Hard constraints

  • Must not regress existing English/Latin caption rendering or increase English-only render time/bundle size.
  • Font files must be properly licensed for redistribution (Noto fonts are OFL — confirm license file/attribution requirements are satisfied, matching how the existing Nunito license is handled if documented).
  • Any new brand.json/Brand type field must be declared in remotion/types/brand.ts and documented per the repo's Agent Implementation Convention.

Dependency issues

  • Depends on the multilingual-pipeline issue only in the sense that end-to-end testing needs real non-English lines.json content — implementation itself can proceed independently using hand-authored sample data.
  • Blocks the bilingual-subtitles issue (dual-language rendering needs both scripts renderable first).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions