Slice 19 — Audio clip sequence: model, planner, composition

.claude/plans/multi-track-audio-breakdown.md

@@ -0,0 +1,96 @@
+# Approved issue breakdown — Multi-track audio timeline
+
+Source: parent PRD issue #1. Approved by Chris on 2026-06-22. Expands PRD out-of-scope items (multi-track audio, media looping). Create in listed order; blocked-by references use real issue numbers.
+
+---
+
+## Slice 19 — Audio clip sequence: model, planner, composition
+
+**Type**: AFK | **Stories**: 17 | **Blocked by**: #34 | **Issue**: #45
+
+**What to build**: Replace the single-soundtrack model with an ordered, sequential playlist of audio clips. Migrate `soundtrackFilename` → `audioClips` on load.
+
+**Acceptance criteria**:
+- [ ] Given multiple audio files in the folder, when the user builds a clip sequence (add / reorder / remove), then `slideshow.json` stores `audioClips: [{ filename, gainDb? }]` and persists across reopen.
+- [ ] Given a clip sequence, when planned, then `RenderPlan` contains ordered audio segments with absolute `startFrame`, `durationInFrames`, `blobUrl`, and per-clip gain (default 0 dB).
+- [ ] Given sequential clips, when played/exported, then clip 2 starts exactly when clip 1 ends (frame-accurate, no gap/overlap).
+- [ ] Given an existing project with only `soundtrackFilename`, when opened, then it loads as a one-clip sequence without data loss.
+
+**Technical notes**: Timeline Core `AudioClip` type; `slidePersistence` migration; `planner.ts` multi-audio; `SlideshowComposition` multiple `<Audio>`; minimal `SoundtrackPanel` clip list. Duration unchanged — `totalFrames` still visual-only.
+
+---
+
+## Slice 20 — Loudness normalization + per-clip gain overrides
+
+**Type**: AFK | **Stories**: new | **Blocked by**: #45 | **Issue**: #47
+
+**What to build**: Pure loudness-analysis module computing recommended gain offset (dB) per audio file. Auto-normalize by default; manual `gainDb` on clip overrides.
+
+**Acceptance criteria**:
+- [ ] Given two synthetic tracks with known RMS difference, when analyzed, then recommended offsets bring perceived levels within ±1 dB of a target.
+- [ ] Given analyzed tracks, when a clip has no manual `gainDb`, then playback/export uses the auto-normalized level.
+- [ ] Given a user-adjusted gain on a clip, when saved, then the manual value wins until reset.
+- [ ] Given analysis results, when the project reopens, then cached offsets in `loudnessCache` avoid re-analysis unless the file changes.
+
+**Technical notes**: `src/audio-analysis/` or extend beat-grid decode; RMS/peak target level; planner `effectiveGainDb`; `soundtrackVolume.ts` combines gain + ducking.
+
+---
+
+## Slice 21 — Audio-driven duration + full-sequence media loop
+
+**Type**: AFK | **Stories**: 17, 18 | **Blocked by**: #45 | **Issue**: #46
+
+**What to build**: When audio exceeds visual timeline, `totalFrames` = sum of clip durations. Loop entire slide sequence deterministically until audio ends.
+
+**Acceptance criteria**:
+- [ ] Given audio longer than one pass of slides, when planned, then `totalFrames` equals total audio duration and slide entries repeat with correct transitions at loop boundaries.
+- [ ] Given audio shorter than visual, when planned, then `totalFrames` equals visual duration.
+- [ ] Given beat sync on, when looping, then nudged durations apply on every pass.
+- [ ] Given a fixed fixture, when planned, then RenderPlan matches a golden snapshot including looped entries.
+
+**Technical notes**: Core planner change; loop boundaries use normal adjacent-slide transitions; videos replay each pass.
+
+---
+
+## Slice 22 — Proportional timeline UI + audio lane with gain controls
+
+**Type**: HITL | **Stories**: 6, 31 | **Blocked by**: #45, #47, #46 | **Issue**: #48
+
+**What to build**: Time-proportional timeline: media thumbnails (width ∝ duration) + audio lane with waveforms, clip boundaries, per-clip gain sliders. Shared scroll; playhead sync.
+
+**Acceptance criteria**:
+- [ ] Given mixed durations, when rendered, then thumbnail widths are proportional to slide duration.
+- [ ] Given an audio clip sequence, when rendered, then clips appear on a lane below media aligned to start times.
+- [ ] Given a gain slider, when adjusted, then playback changes immediately and `gainDb` persists.
+- [ ] Given playback or scrubbing, when the playhead moves, then media and audio lanes highlight consistently.
+- [ ] HITL: owner signs off on iMovie-like usability on a 10-clip project.
+
+**Technical notes**: `TimelinePanel.tsx` or refactor `StoryboardFilmstrip`; composition-patterns skill; Remotion waveform or Web Audio peaks.
+
+---
+
+## Slice 23 — Beat grid across multi-clip audio timeline
+
+**Type**: AFK | **Stories**: 18–22 | **Blocked by**: #45, #46 | **Issue**: #49
+
+**What to build**: Beat sync against concatenated virtual soundtrack. Per-file `beatGridCache`; effective grid spans full timeline with clip start offsets.
+
+**Acceptance criteria**:
+- [ ] Given 2+ clips, when beat sync on, then boundaries nudge against beats on combined timeline.
+- [ ] Given manual beat grid, when clips reordered, then beat positions stay correct relative to concatenated playback.
+- [ ] Given beat sync with looped media, when planned, then nudging aligns across full audio duration.
+- [ ] Given per-file cache, when a new clip added, then only that file is analyzed.
+
+**Technical notes**: `concatBeatGrid` pure function; `useBeatGrid` on clip list changes.
+
+---
+
+## Published
+
+| Slice | Issue |
+|-------|-------|
+| 19 — Audio clip sequence | #45 |
+| 20 — Loudness normalization | #47 |
+| 21 — Audio-driven duration + loop | #46 |
+| 22 — Proportional timeline UI | #48 |
+| 23 — Beat grid multi-clip | #49 |

specs/multi-track-audio-timeline.md

@@ -0,0 +1,40 @@
+# Multi-track audio timeline
+
+Parent issues: #45–#49. This spec tracks slice 19 (#45) implementation.
+
+## Slice 19 — Audio clip sequence (done)
+
+Replace single `soundtrackFilename` with ordered `audioClips[]`. Sequential playback in RenderPlan. `totalFrames` remains visual-only.
+
+Closes #45.
+
+### Data model
+
+```typescript
+type AudioClip = { filename: string; gainDb?: number }
+type SerializedAudioClip = { filename: string; gainDb?: number }
+```
+
+Migration: `soundtrackFilename` → single `audioClips` entry on load.
+
+### RenderPlan
+
+```typescript
+type AudioSegment = {
+  blobUrl: string
+  durationInFrames: number
+  gainDb: number
+  startFrame: number
+}
+
+type RenderPlan = {
+  entries: RenderPlanEntry[]
+  audioSegments?: AudioSegment[]
+  duckingEnvelope?: DuckingEnvelope
+  totalFrames: number
+}
+```
+
+### UI (minimal)
+
+SoundtrackPanel: ordered clip list, add / reorder / remove. Beat grid uses first clip until #49.

src/composition/SlideshowComposition.tsx

@@ -1,15 +1,15 @@
 import React, { useCallback } from 'react'
-import { AbsoluteFill, Img, interpolate, useCurrentFrame } from 'remotion'
+import { AbsoluteFill, Img, interpolate, Sequence, useCurrentFrame } from 'remotion'
 import { Audio, Video } from '@remotion/media'
 import { TransitionSeries, linearTiming } from '@remotion/transitions'
 import { fade } from '@remotion/transitions/fade'
 import type { TransitionPresentation, TransitionPresentationComponentProps } from '@remotion/transitions'
 import { loadFont } from '@remotion/google-fonts/Inter'
-import type { RenderPlanEntry, RenderPlan } from '../sequence-planner/types'
+import type { AudioSegment, RenderPlanEntry, RenderPlan } from '../sequence-planner/types'
 import type { TransitionType } from '../timeline-core/settings'
 import { isTitleSlide } from '../timeline-core/types'
 import type { MediaSlide, TitleSlide } from '../timeline-core/types'
-import { volumeAtFrame } from './soundtrackVolume'
+import { dbToLinear, volumeAtFrame } from './soundtrackVolume'
 
 type MediaRenderPlanEntry = Omit<RenderPlanEntry, 'slide'> & { slide: MediaSlide }
 type TitleRenderPlanEntry = Omit<RenderPlanEntry, 'slide'> & { slide: TitleSlide }
@@ -62,8 +62,8 @@ export function SlideshowComposition({ plan }: SlideshowProps) {
 
   return (
     <AbsoluteFill>
-      {plan.soundtrack ? (
-        <SoundtrackAudio track={plan.soundtrack} />
+      {plan.audioSegments && plan.duckingEnvelope ? (
+        <AudioSegments duckingEnvelope={plan.duckingEnvelope} segments={plan.audioSegments} />
       ) : null}
       <TransitionSeries>
       {plan.entries.map((entry) => (
@@ -89,13 +89,39 @@ export function SlideshowComposition({ plan }: SlideshowProps) {
 }
 
 // @remotion/media's Audio (not Html5Audio) — required for renderMediaOnWeb export.
-function SoundtrackAudio({ track }: { track: NonNullable<RenderPlan['soundtrack']> }) {
-  const { duckingEnvelope } = track
+function AudioSegments({
+  duckingEnvelope,
+  segments,
+}: {
+  duckingEnvelope: NonNullable<RenderPlan['duckingEnvelope']>
+  segments: AudioSegment[]
+}) {
+  return segments.map((segment, index) => (
+    <Sequence
+      durationInFrames={segment.durationInFrames}
+      from={segment.startFrame}
+      key={`${segment.startFrame}-${segment.blobUrl}-${index}`}
+      layout="none"
+    >
+      <AudioSegmentAudio duckingEnvelope={duckingEnvelope} segment={segment} />
+    </Sequence>
+  ))
+}
+
+function AudioSegmentAudio({
+  duckingEnvelope,
+  segment,
+}: {
+  duckingEnvelope: NonNullable<RenderPlan['duckingEnvelope']>
+  segment: AudioSegment
+}) {
+  const gainLinear = dbToLinear(segment.gainDb)
   const volume = useCallback(
-    (frame: number) => volumeAtFrame(duckingEnvelope, frame),
-    [duckingEnvelope],
+    (localFrame: number) =>
+      volumeAtFrame(duckingEnvelope, segment.startFrame + localFrame) * gainLinear,
+    [duckingEnvelope, gainLinear, segment.startFrame],
   )
-  return <Audio src={track.blobUrl} volume={volume} />
+  return <Audio src={segment.blobUrl} volume={volume} />
 }
 
 function TitleSlideView({ entry }: { entry: TitleRenderPlanEntry }) {

src/composition/soundtrackVolume.ts

@@ -1,6 +1,10 @@
 import { interpolate } from 'remotion'
 import type { DuckingEnvelope } from '../sequence-planner/types'
 
+export function dbToLinear(gainDb: number): number {
+  return Math.pow(10, gainDb / 20)
+}
+
 export function volumeAtFrame(envelope: DuckingEnvelope, frame: number): number {
   const { keyframes, rampFrames } = envelope
   if (keyframes.length === 0) return 1

src/editor-shell/App.tsx

@@ -37,16 +37,17 @@ export function App() {
   const {
     aspectRatio,
     setAspectRatio,
+    audioClips,
     audioTracks,
     beatGridCache,
     globalSettings,
     setGlobalSettings,
     manualBeatGrid,
     slides,
     setSlides,
-    soundtrackFilename,
     themeName,
     setThemeName,
+    updateAudioClips,
     updateBeatGridPersist,
     loading,
     error,
@@ -56,11 +57,13 @@ export function App() {
     recentProjects,
   } = project
 
+  const primaryClipFilename = audioClips[0]?.filename ?? null
+
   const beatGrid = useBeatGrid({
     audioTracks,
     onPersistChange: updateBeatGridPersist,
     persisted: { beatGridCache, manualBeatGrid },
-    soundtrackFilename,
+    primaryClipFilename,
   })
 
   const handleReorder = useCallback((fromIndex: number, toIndex: number) => {
@@ -99,27 +102,28 @@ export function App() {
     setSlides(prev => prev.map(s => (s.id === id && isTitleSlide(s) ? { ...s, ...updates } : s)))
   }, [setSlides])
 
-  const selectedSoundtrack = useMemo(
-    () => (soundtrackFilename
-      ? audioTracks.find((track) => track.filename === soundtrackFilename)
-      : undefined),
-    [audioTracks, soundtrackFilename],
+  const planAudioClips = useMemo(
+    () => audioClips.map((clip) => {
+      const track = audioTracks.find((entry) => entry.filename === clip.filename)
+      if (!track) return null
+      return {
+        blobUrl: track.blobUrl,
+        durationInFrames: track.durationInFrames,
+        ...(clip.gainDb !== undefined ? { gainDb: clip.gainDb } : {}),
+      }
+    }).filter((clip) => clip !== null),
+    [audioClips, audioTracks],
   )
 
   const renderPlan = useMemo(
     () => plan(
       filterIncluded(slides),
       globalSettings,
       undefined,
-      selectedSoundtrack
-        ? {
-            blobUrl: selectedSoundtrack.blobUrl,
-            durationInFrames: selectedSoundtrack.durationInFrames,
-          }
-        : undefined,
+      planAudioClips.length > 0 ? planAudioClips : undefined,
       beatGrid.effectiveBeatGrid,
     ),
-    [beatGrid.effectiveBeatGrid, globalSettings, selectedSoundtrack, slides],
+    [beatGrid.effectiveBeatGrid, globalSettings, planAudioClips, slides],
   )
   const totalFrames = renderPlan.totalFrames > 0 ? renderPlan.totalFrames : FPS
   const canvas = dimensionsForAspectRatio(aspectRatio)
@@ -220,10 +224,10 @@ export function App() {
                 onClearManualBeatGrid={beatGrid.clearManualBeatGrid}
                 onJamendoAdd={project.addJamendoTrack}
                 onSettingsChange={handleSettingsChange}
-                onSoundtrackChange={project.changeSoundtrack}
+                onAudioClipsChange={updateAudioClips}
                 onThemeChange={handleThemeChange}
                 settings={globalSettings}
-                soundtrackFilename={soundtrackFilename}
+                audioClips={audioClips}
                 themeName={themeName}
               />
             }

src/editor-shell/EditorSidebar.tsx

@@ -6,6 +6,7 @@ import {
 } from '@/components/ui/accordion'
 import { Button } from '@/components/ui/button'
 import type { AspectRatio, GlobalSettings, ThemeName } from '../timeline-core'
+import type { AudioClip } from '../timeline-core/types'
 import type { AudioTrack } from '../project-store'
 import type { BeatGrid } from '../beat-grid/types'
 import type { JamendoAttribution, JamendoTrack } from '../jamendo/types'
@@ -25,13 +26,13 @@ type Props = {
   onApplyManualBpm: (bpm: number, firstBeatOffsetSecs: number) => void
   onApplyTapTimestamps: (tapTimestampsMs: number[]) => void
   onAspectRatioChange: (ratio: AspectRatio) => void
+  onAudioClipsChange: (clips: AudioClip[]) => void
   onClearManualBeatGrid: () => void
   onJamendoAdd: (track: JamendoTrack, attribution: JamendoAttribution) => Promise<void>
   onSettingsChange: (updated: GlobalSettings) => void
-  onSoundtrackChange: (filename: string | null) => void
   onThemeChange: (name: ThemeName) => void
+  audioClips: AudioClip[]
   settings: GlobalSettings
-  soundtrackFilename: string | null
   themeName: ThemeName | null
 }
 
@@ -46,13 +47,13 @@ export function EditorSidebar({
   onApplyManualBpm,
   onApplyTapTimestamps,
   onAspectRatioChange,
+  onAudioClipsChange,
   onClearManualBeatGrid,
   onJamendoAdd,
   onSettingsChange,
-  onSoundtrackChange,
   onThemeChange,
+  audioClips,
   settings,
-  soundtrackFilename,
   themeName,
 }: Props) {
   return (
@@ -85,15 +86,15 @@ export function EditorSidebar({
               <AccordionContent>
                 <SoundtrackPanel
                   analysisStatus={analysisStatus}
+                  audioClips={audioClips}
                   audioTracks={audioTracks}
                   beatSync={settings.beatSync !== false}
                   effectiveBeatGrid={effectiveBeatGrid}
                   manualBeatGrid={manualBeatGrid}
                   onApplyManualBpm={onApplyManualBpm}
                   onApplyTapTimestamps={onApplyTapTimestamps}
-                  onChange={onSoundtrackChange}
+                  onChange={onAudioClipsChange}
                   onClearManualBeatGrid={onClearManualBeatGrid}
-                  soundtrackFilename={soundtrackFilename}
                 />
               </AccordionContent>
             </AccordionItem>