[Feat] feature-demo skill: polished, narrated demo videos - #1177
Open
mrubens wants to merge 11 commits into
Open
[Feat] feature-demo skill: polished, narrated demo videos#1177mrubens wants to merge 11 commits into
mrubens wants to merge 11 commits into
Conversation
A new standard skill that produces Screen Studio-style feature demos: the demo script drives real browser interactions (delegated to the proof-runner subagent via agent-browser) while logging cursor, click, and element-rect timelines on the same clock, then a bundled Remotion composition post-produces the recording with zoom-to-element moves, click ripples, captions, and optional voice-over narration. - packages/cloud-agents .../feature-demo: SKILL.md orchestration, capture runner, narration + timing-fit scripts, render project (excluded from tsc/knip/oxlint like agent-browser; copied verbatim into the sandbox skills dir at activation) - apps/api /api/tts/narration: control-plane TTS endpoint (task-token policy + rate limit). ElevenLabs credentials live only on the control plane; R_ELEVENLABS_* are control-plane env names, stripped from sandbox env injection. Unconfigured deployments 404 and the skill degrades to captions-only. - apps/worker/Dockerfile: bake Remotion's chrome-headless-shell at /opt/remotion/headless-shell (the bundled Chrome removed old headless); runtime fallback via remotion browser ensure for older snapshots. - docs: cookbook recipe + environment variable reference.
Contributor
|
1 issue outstanding. See task
Reviewed b3175e1 |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+52
to
+54
| const js = `(function(){var e=document.querySelector(${JSON.stringify( | ||
| sel, | ||
| )});if(!e)return null;var r=e.getBoundingClientRect();return{x:r.x,y:r.y,w:r.width,h:r.height};})()`; |
Contributor
Author
There was a problem hiding this comment.
Addressed in b995257: the selector now travels into the page-eval as base64 with a strict alphabet check before interpolation, so no selector bytes can reach the constructed code.
![roomote-community[bot]](https://avatars.githubusercontent.com/u/211522643?s=60&v=4){kind=small}
eleven_v3 reads slowly and needs the 1.12 atempo speed-up; v2-family models pace naturally and should not be sped up. The narration endpoint now returns its modelId, build-narration records it in the manifest, and fit-timing derives the atempo default from it (argv still wins).
Per review direction: the bundled Remotion project is a starting point the agent copies and adapts (branding, captions, layout, presets), not a fixed pipeline stage. The timeline JSON stays the stable contract between capture and render. When adapting, the skill installs Remotion's official agent skills (remotion-dev/skills — their sanctioned successor to the deprecated Remotion MCP) for current API guidance. A render/ README documents the bug classes worth preserving (counter-scaled cursor, edge-clamp guard, timeline-driven interpolation).
Simplify per review: one model, one settings profile, no R_ELEVENLABS_MODEL knob and no model-aware atempo plumbing. multilingual_v2 reads voice clones naturally at a natural pace; the timing fit's atempo stays available as an explicit argv override.
- capture: bracket cursor motion with a hold key at motion start so the synthetic cursor no longer drifts toward its next target through waits, holds, and scrolls (and clicks on a fresh element get a short bracketed hop instead of a slow glide). - capture: pass selectors into the page-eval as alphabet-checked base64 so no selector bytes are interpolated into code (CodeQL). - fit-timing: solve the playback rate by simulating the schedule instead of a closed-form cumulative estimate; a line that waits for a late anchor ends later than the estimate, which could push later lines past their anchors. Lines intentionally START just before their zoom and play over its hold; the docs now say so explicitly.
The cookbook index is generated from recipe frontmatter; run the generator instead of hand-editing between the markers.
The binary name differs per platform: headless_shell on linux-arm64, chrome-headless-shell on linux64, so the hardcoded name failed the amd64 image build. Resolve it with a find over both names (validated under amd64 emulation and in the arm64 sandbox image).
Capture emits an explicit per-caption anchor (the zoom's landing time); the timing fit schedules against it. caption.start begins with the cursor glide ~moveMs earlier, so anchoring there started narration before the zoom even began moving. Also hoist the Dockerfile block comment above the RUN chain.
Field failure from a real game demo: CDP screencast only emits frames on visual damage and stamps them without wall-clock gaps, so a 52s interaction over a static-rendering surface collapsed into a sub-second video. The capture runner now injects an imperceptible 2px rAF ticker after record start so compositor damage flows at wall-clock rate (static-page regression: 7.93s timeline -> 8.0s recording), closes any pre-existing pages before recording (record start creates a fresh browser context; a page left over from inspection would split beats and recording across contexts), and fails loudly when the recording comes back much shorter than the interaction instead of letting a garbage video flow downstream. SKILL.md routes that failure to a 'stale sandbox runtime' blocker (old snapshots carry a 2018 legacy ffmpeg that crashes record stop under real frame volume).
Contributor
Author
Field-failure fix: static surfaces recorded as sub-second videosA real dogfood run (game demo) surfaced a capture failure mode none of the development surfaces hit: CDP screencast only emits frames on visual damage, so a statically-rendered app (games between state changes, idle dashboards) collapsed a 52-second interaction into a 0.8s video with no wall-clock mapping. Docs pages and fixtures masked this because scrolling and hover repaint constantly. Fixed in 8dc12b2, all reproduced/verified against live sandboxes:
|
Second dogfood run failed at the delegation seam in two ways: the brief's home-relative runner path did not resolve in the delegated runtime, and the proof-runner refused the capture script as a disallowed browser path (its instructions allow only the agent-browser CLI, and a per-brief assurance is not reliable authority). - The skill now stages the runner at /tmp/feature-demo/capture.mjs before delegating (no home-relative paths across the boundary) and parent-side paths use $HOME explicitly. - The proof-runner prompt carries a narrow standing exception: the staged runner is an agent-browser orchestrator and running it as the brief specifies is compliant; the exception names that one path and no other script.
Contributor
Author
Second dogfood failure: the delegation seamA second live run failed before capture in two ways, both at the parent→proof-runner boundary:
Fixed in b17b6c4 with prompt/skill test coverage for both. |
| '- Before the first browser command, explicitly load the `agent-browser` skill or CLI-served guidance exactly once. If the OpenCode Skill tool is available, invoke the `agent-browser` skill. If it is not available, run `agent-browser skills get core --full` in the shell and treat that output as the browser usage guide.', | ||
| '- `agent-browser` is a command-line executable, not an OpenCode tool or MCP tool. Invoke it with shell commands such as `agent-browser get url`; do not look for an internal tool named `agent-browser`.', | ||
| '- `agent-browser` is the only allowed browser automation CLI. Do not use Playwright, browser DevTools, curl-only screenshot substitutes, or any other browser automation path.', | ||
| '- One sanctioned exception: the feature-demo capture runner at `/tmp/feature-demo/capture.mjs`, staged there by the parent. It is an `agent-browser` orchestrator — every browser action it performs goes through the `agent-browser` CLI — and running it via `node` with the environment variables the brief specifies is compliant browser work, not a disallowed automation path. Its outputs land where the brief says (typically `/tmp/feature-demo/work/`), not under `/tmp/capture-visual-proof/`. This exception covers exactly that staged runner and no other script.', |
Contributor
There was a problem hiding this comment.
This exception trusts only a parent-writable /tmp pathname. Any parent task can stage a different capture.mjs at that path and ask the proof runner to execute it, so this turns the intended narrow agent-browser allowance into arbitrary Node execution in the proof-runner context. Stage a trusted immutable copy in the runner's own runtime or verify the staged file against a trusted digest before invoking it.
Live-sandbox controls showed the sparse-frames failure is page-specific: a WebGL game stalls headless frame production for the whole page while plain pages on the same sandbox record at full wall-clock rate. SKILL.md now names that blocker (webgl surface stalls headless recording) so agents diagnose by target instead of burning retries; the stale-runtime ffmpeg hint stays scoped to actual record-stop ffmpeg errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new
feature-demostandard skill that produces polished, Screen Studio-style demo videos of product features: a real recording driven live in the sandbox browser, post-produced with smooth zoom-to-element moves, a synthetic cursor, click ripples, captions, and optional voice-over narration.Invoke with
$feature-demoor by asking for a demo video in a task.How it works
One declarative demo script drives everything. The capture runner performs the real browser interactions (scroll, hover, click, type) via the
agent-browserCLI and records, on the same clock, where the cursor is, when clicks land, and the resolved rectangle of every focused element. The bundled Remotion composition consumes that timeline, so a zoom can never drift from the element it targets.Pipeline: author script → capture (delegated to the
proof-runnersubagent, keeping browser containment intact) → narrate (optional) → fit timing → render → verify → upload viamanage_artifacts.The bundled Remotion project is a reference template, not a fixed pipeline stage: the agent copies it into the work dir and adapts the copy (branding, caption styling, layout, presets) when a demo calls for it, installing Remotion's official agent skills (
remotion-dev/skills— their sanctioned successor to the deprecated Remotion MCP) for API guidance. The timeline JSON is the stable contract between capture and render, and arender/README.mddocuments the hard-won invariants worth preserving (counter-scaled cursor, edge-clamp guard, timeline-driven interpolation).The timing fit encodes what makes these demos feel produced rather than recorded: the dead opening hold is trimmed, the voice-over is paced into a conversational band (pitch-preserving
atempo), and the recording's playback rate is solved so each spoken line finishes just before its zoom lands.Narration without the key in the sandbox
Narration uses ElevenLabs, but the provider key never enters a task sandbox:
POST /api/tts/narration(route policytask-token, rate-limited, streamed body cap). The sandbox posts plain caption text with its run-scoped token; the control plane holdsR_ELEVENLABS_API_KEY/R_ELEVENLABS_VOICE_IDand returns audio. Same posture as the inference gateway and custom MCP proxy.R_ELEVENLABS_*are registered as control-plane env names, so they are reserved from the generic env editor and stripped from sandbox env injection.Worker image
Remotion cannot render with the bundled agent-browser Chrome (Chrome 143 removed old-headless), so the image now bakes Remotion's
chrome-headless-shellat/opt/remotion/headless-shell(both architectures, pinned, arch-independent symlink). Older snapshots fall back tonpx remotion browser ensureat render time — the smoke run below exercised exactly that fallback.Testing
pnpm lint,pnpm check-types,pnpm knipgreen across workspaces.npm install+ headless-shell fallback → rendered a clean 7.6s 1080p mp4, verified with ffprobe and frame inspection.Docs
environment-variablesreference for the two new optional vars, including the recommendation to scope the ElevenLabs key to text-to-speech only.