Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ public/thumbnail/ # candidate frames, cutouts, manifest
public/renders/ # final rendered .mp4 files
public/output/ # carousel exports
public/shorts/
public/line-captions/
public/sync/output

# ── Editable pipeline outputs (text/JSON) ─────────────────────────────────────
Expand Down
28 changes: 28 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,27 @@ segments[]

`CropViewport`: `cx/cy` = normalised centre (0–1), `w/h` = crop dimensions (0–1).

### lines.json

Standalone artifact for the line-caption shorts pipeline (`public/line-captions/{id}/lines.json`) — not part of `transcript.json`; no segments/tokens/cuts.

```
meta
title: string
duration: number total video duration in seconds
fps: 60
videoSrc?: string path relative to /public

lines[]
id: number sequential across the whole clip
speaker: string
text: string exactly wordsPerLine (default 3) whole words
startMs: number milliseconds from start of source video
endMs: number
```

A line never spans a segment/speaker boundary. Produced by `chunkIntoLines()` (`scripts/line-captions/chunkLines.js`); human-edited via `lines.doc.txt` (`captions:create` / `captions:merge`) which only rewrites `text` — `startMs`/`endMs`/`speaker` are untouched by edits.

---

## Rendering Model
Expand Down Expand Up @@ -214,6 +235,13 @@ ty = (0.5 - vp.cy) × 100%
| `app/editor/Timeline.tsx` | Timeline component (630 lines) | Decompose + add waveform (Phase 8) |
| `app/context/AuthContext.tsx` | Auth context with hardcoded credentials | Fix (Phase 7) |
| `vscode-transcript-language/src/extension.js` | VSCode transcript extension | Add Wrap-in-cut command (Phase 0.5) |
| `remotion/types/lineCaptions.ts` | `CaptionLine`, `LineCaptionsMeta`, `LineCaptionsDoc` | — |
| `remotion/components/LineCaptionOverlay.tsx` | Renders the active fixed-window `CaptionLine`; speaker-tinted via `brand.colors.palette` | — |
| `remotion/LineCaptionClip.tsx` | Raw video pass-through + `LineCaptionOverlay`; registered per clip in `Root.tsx` as `LineCaptionClip-{id}` | — |
| `scripts/line-captions/chunkLines.js` | `chunkIntoLines()` — merges BPE tokens into whole words, buckets into fixed-size (default 3) caption lines, never spanning a segment/speaker boundary | — |
| `scripts/line-captions/create-line-captions.js` | Raw portrait video → audio extract → transcribe → align → diarize (if `--num-speakers > 1`) → `lines.json` + `lines.doc.txt` | — |
| `scripts/line-captions/merge-line-captions.js` | Re-parses `lines.doc.txt`, overwrites `text` on matching `lines.json` entries by `[id]` | — |
| `scripts/line-captions/render-line-captions.js` | Renders `LineCaptionClip-{id}` to `public/renders/` | — |

---

Expand Down
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -443,6 +443,59 @@ Output MP4 is written to the path stored in `transcript.meta.outName`.

---

## Editing: Line-caption shorts

A standalone pipeline for a raw short-form portrait video that just needs burned-in captions — no camera cuts, no hooks. Each clip lives in `public/line-captions/<clip-id>/`.

```
raw portrait video → captions:create → captions:merge → captions:render
```

### 1. Create captions

```bash
npm run captions:create -- --video path/to/clip.mp4
npm run captions:create -- --video path/to/clip.mp4 --num-speakers 2
npm run captions:create -- --video path/to/clip.mp4 --id mediocrity
```

Extracts audio, transcribes, aligns word timestamps, diarizes (if `--num-speakers` > 1), and chunks the transcript into 3-word caption lines. `--id` picks the clip slug; omit it to auto-increment `clip-1`, `clip-2`, etc.

Output: `public/line-captions/<clip-id>/lines.json` and `lines.doc.txt`.

### 2. Edit the doc

Open `lines.doc.txt`. Lines are grouped under `=== SPEAKER ===` headers:

```
=== Natasha ===

[1] And I think the
[2] real issue is context
```

Reword lines as needed — retype the text after each `[id]` marker. Don't add, remove, or reorder the `[id]` markers themselves; timing stays tied to the original id.

### 3. Apply edits

```bash
npm run captions:merge -- --id mediocrity
```

Writes your edited wording back into `lines.json`. `startMs`/`endMs`/`speaker` are left untouched.

### 4. Render

```bash
npm run captions:render -- --id mediocrity
```

Renders the source video with burned-in captions via the `LineCaptionClip-<clip-id>` Remotion composition. Output: `public/renders/<clip-id>.mp4`.

You can also preview first in `npx remotion studio` — select `LineCaptionClip-<clip-id>` — before rendering.

---

## Pipeline: Carousel Generation

```bash
Expand Down
Loading
Loading