Skip to content

feat: add Gemini agentic video adapter - #1608

Draft
groupthinking wants to merge 2 commits into
mainfrom
feat/gemini-agentic-video
Draft

feat: add Gemini agentic video adapter#1608
groupthinking wants to merge 2 commits into
mainfrom
feat/gemini-agentic-video

Conversation

@groupthinking

Copy link
Copy Markdown
Owner

What changed

Adds an opt-in adapter for Google's agentic video understanding path via the Interactions API.

  • accepts YouTube URLs and file URIs
  • supports agentic and static processing per video, including mixed requests
  • defaults to gemini-3.7-flash
  • returns a durable execution receipt with model, sources, processing modes, text, and token count
  • validates inputs before a provider call

Why

Google AI Studio announced agentic video understanding on September 1, 2026. The capability actively inspects transcript, frames, and audio with adaptive frame sampling, which maps directly to EventRelay's video-to-event pipeline and the See → Act → Record → Review loop.

Sources:

Verification

  • python -m compileall src/integration/gemini_agentic_video.py tests/unit/test_gemini_agentic_video.py — passed
  • dependency-free fake-provider harness — passed request construction, async invocation, receipt, and validation
  • full pytest was not available in the local execution environment (No module named pytest); repository CI remains the merge gate

No production deployment or default-path switch is included.

@vercel

vercel Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
v0-uvai Canceled Canceled v0 Sep 2, 2026 7:45am UTC

@coderabbitai

coderabbitai Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the python label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 18656dc.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

Copy link
Copy Markdown
Owner Author

Execution receipt — security workflow diagnosis

Verified failing run: Security Scan 33605167703, job trivy (100167379816).

The container build stops before Trivy executes because:

npm ci --workspace=apps/web --production --legacy-peer-deps
Invalid: lock file's @ai-sdk/gateway@4.0.23 does not satisfy @ai-sdk/gateway@4.0.72
Invalid: lock file's postcss@8.5.21 does not satisfy postcss@8.5.26
ERROR: process ... did not complete successfully: exit code: 1

The PR changes only:

  • src/integration/gemini_agentic_video.py
  • tests/unit/test_gemini_agentic_video.py

Interpretation: the failed security workflow is a repository package/lock synchronization blocker, not a vulnerability finding from this adapter. Trivy was skipped because the image never built.

Action: recorded here rather than mixing an unrelated lockfile rewrite into this focused PR. The next safe repair is to reconcile the web workspace lockfile in its canonical dependency-maintenance work, then rerun Security Scan.

Other exact-head checks currently verified successful: CI, Coverage, CodeQL, Secret Scan, and Dependency Review.

Copy link
Copy Markdown
Owner Author

Execution receipt — Gemini 3.8 adoption boundary and benchmark scope

New verified fact (2026-09-02): Google released Gemini 3.8 Flash, and Vercel AI Gateway exposes it as google/gemini-3.8-flash with video input. Main now uses that gateway path in 25e6fbb.

Separate verified fact: Google's current agentic-video guide still documents processing: "agentic" only for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite. It does not document Gemini 3.8 for this specialized Interactions API mode.

Sources:

Decision: TEST. Keep this draft's default at gemini-3.7-flash until Google explicitly confirms 3.8 support for processing: "agentic". Do not treat the generic gateway video path and the Interactions API agentic path as equivalent.

Re-scoped value of this PR: retain it as the explicit agentic-processing/receipt adapter and use it to benchmark against the main gateway extractor on the same videos:

  1. Video Pack schema completeness and grounded timestamps
  2. Token use and actual cost
  3. Latency
  4. Failure behavior
  5. Provenance/receipt quality

The adapter already permits a model override through GEMINI_AGENTIC_VIDEO_MODEL, so no speculative model-pin change is needed today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant