Skip to content

fix: route every transcript client through the centralized proxy (#1087) #1129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
groupthinking wants to merge 3 commits into from
Closed

fix: route every transcript client through the centralized proxy (#1087) #1129

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b6f4cf3
fix: route every transcript client through the centralized proxy (#1087)
groupthinking Jul 31, 2026
5659f2b
fix(proxy): resolve helper standalone instead of silently disabling p…
groupthinking Jul 31, 2026
49e887c
test(proxy): reject opaque and foreign proxy configs
groupthinking Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion src/agents/interactive_metadata_extractor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,25 @@
from dotenv import load_dotenv
from youtube_transcript_api import YouTubeTranscriptApi

try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError: # pragma: no cover - standalone execution outside the package
import sys as _sys
from pathlib import Path as _Path

# Running this module by path (the documented CLI entry point) leaves the
# repository's ``src`` directory off sys.path. Bootstrap it so the canonical
# helper resolves instead of silently disabling the proxy.
_sys.path.insert(0, str(_Path(__file__).resolve().parents[1]))
try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError as _exc: # pragma: no cover - helper genuinely unreachable
raise ImportError(
"youtube_extension.utils.proxy is required so transcript requests "
"honour WEBSHARE_PROXY_URL; refusing to continue with unproxied "
"egress."
) from _exc

load_dotenv()
logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -84,7 +103,10 @@ async def extract_transcript(self, video_id: str) -> list[dict[str, Any]]:
# event loop free.
loop = asyncio.get_event_loop()
transcript = await loop.run_in_executor(
None, lambda: YouTubeTranscriptApi().fetch(video_id).to_raw_data()
None,
lambda: YouTubeTranscriptApi(proxy_config=get_transcript_proxy_config())
.fetch(video_id)
.to_raw_data(),
Comment on lines +106 to +109

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift

Stop returning fabricated transcript results.

Both paths report successful extraction after real extraction fails. InteractiveMetadataExtractor.extract_transcript falls back to _generate_transcript_whisper, which returns a hard-coded segment. RealVideoProcessor._extract_transcript_with_rotation returns a fixed segment after yt_dlp.extract_info() without extracting subtitle content. This sends false video data to downstream agents.

Use a real transcriber and validate its output. If no real transcript is available, raise a typed extraction error.

  • src/agents/interactive_metadata_extractor.py#L106-L109: Do not call the placeholder transcript fallback after this request fails.
  • src/agents/process_video_with_mcp.py#L241-L246: Propagate the direct extraction failure when no real fallback succeeds.
  • src/agents/process_video_with_mcp.py#L257-L260: Replace the fixed yt_dlp fallback result with actual subtitle extraction or an error.

As per coding guidelines, “Production code must use real behavior only: no mock delays, fake data, or simulated responses.” As per path instructions, “Flag any file that contains placeholder/stub implementations … or returns mock/fake data.”

📍 Affects 2 files
  • src/agents/interactive_metadata_extractor.py#L106-L109 (this comment)
  • src/agents/process_video_with_mcp.py#L241-L246
  • src/agents/process_video_with_mcp.py#L257-L260
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agents/interactive_metadata_extractor.py` around lines 106 - 109, Remove
fabricated transcript fallbacks and ensure only validated real transcript data
is returned. In src/agents/interactive_metadata_extractor.py lines 106-109, stop
invoking the placeholder _generate_transcript_whisper fallback after the YouTube
request fails. In src/agents/process_video_with_mcp.py lines 241-246, propagate
the direct extraction failure when no real fallback succeeds. In
src/agents/process_video_with_mcp.py lines 257-260, replace the fixed result in
RealVideoProcessor._extract_transcript_with_rotation with actual subtitle
extraction, or raise a typed extraction error when unavailable.

Sources: Coding guidelines, Path instructions

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Locate every transcript client construction and any existing bounded Session wrapper.
rg -n -C 3 --glob='*.py' 'YouTubeTranscriptApi\s*\(' src
rg -n -C 3 --glob='*.py' 'http_client\s*=|class .*Session|def request\(' src

# Confirm the declared youtube-transcript-api constraint.
sed -n '1,220p' pyproject.toml

Repository: groupthinking/EventRelay

Length of output: 16595

🌐 Web query:

youtube-transcript-api http_client YouTubeTranscriptApi __init__ _http_client requests Session timeout

💡 Result:

In the python library youtube-transcript-api, you can control the HTTP configuration by passing a custom requests.Session object to the YouTubeTranscriptApi constructor via the http_client parameter [1][2]. While the library does not have a native timeout parameter, you can manage request timeouts by configuring the Session object before passing it to the API [1][2][3]. To set a timeout for all requests made by the library: 1. Initialize a requests.Session object. 2. Manually set a default timeout for the session's request methods or use a transport adapter. A common pattern in the requests library is to monkeypatch the session or override the request method, as requests.Session does not have a built-in default timeout attribute [3]. Example of passing a custom session: from requests import Session from youtube_transcript_api import YouTubeTranscriptApi # Create a session http_client = Session # Customize the session (e.g., custom headers) http_client.headers.update({"Accept-Encoding": "gzip, deflate"}) # Inject the session into the API ytt_api = YouTubeTranscriptApi(http_client=http_client) ytt_api.fetch(video_id) Note that YouTubeTranscriptApi is not thread-safe because it uses a requests.Session object internally [2][4]. If you are working in a multi-threaded environment, you must initialize a new instance of YouTubeTranscriptApi (and ideally a new requests.Session) per thread [2][4]. Regarding the timeout specifically, users have noted that the library lacks a direct timeout argument, and requests for this feature have been discussed in the project's issue tracker [3]. Users are often advised to handle timeouts by wrapping the API calls or customizing the underlying requests session [3].

Citations:

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Inspect the executor call sites and whether any youtube-transcript-api call
# passes http_client or configures timeouts.
rg -n -C 4 'http_client|timeout' src/agents src/integration/youtube_api.py src/mcp/mcp_video_processor.py src/youtube_extension/backend/enhanced_video_processor.py src/youtube_extension/backend/services/youtube/adapters 2>/dev/null || true

# Static model of YouTubeTranscriptApi construction semantics by reading the installed package
# if available, without network access or repo execution.
python3 - <<'PY'
import inspect, importlib.util
mods = ["youtube_transcript_api._api", "youtube_transcript_api"]
for mod_name in mods:
    spec = importlib.util.find_spec(mod_name)
    print(mod_name, "FOUND" if spec else "NOT_FOUND", getattr(spec, "origin", None))
    if spec:
        mod = importlib.util.module_from_spec(spec)
        try:
            spec.loader.exec_module(mod)
        except Exception as exc:
            print("LOAD_ERROR", type(exc).__name__, str(exc))
            continue
        if hasattr(mod, "YouTubeTranscriptApi"):
            print("YouTubeTranscriptApi file:", getattr(mod.YouTubeTranscriptApi, "__module__", None))
            print(inspect.signature(mod.YouTubeTranscriptApi.__init__))
            src = inspect.getsource(mod.YouTubeTranscriptApi.__init__)
            print(src)
PY

Repository: groupthinking/EventRelay

Length of output: 40895

Set a finite HTTP timeout for every transcript client.

youtube-transcript-api accepts a custom HTTP client, but these calls pass only proxy_config. Without an enforced connect/read timeout, a blocked transcript/SRT fetch can hold an executor worker and stall the surrounding timeout or circuit-breaker path.

Add one shared transcript HTTP-client factory that creates a requests.Session with bounded connect and read timeouts, and pass it through http_client= at every transcript API call, including:

  • src/agents/interactive_metadata_extractor.py#L106-L109
  • src/agents/process_video_with_mcp.py#L241-L246
  • src/agents/process_video_with_mcp.py#L257-L260
  • src/integration/youtube_api.py#L118-C20
  • src/mcp/mcp_video_processor.py#L713
  • src/mcp/mcp_video_processor.py#L737

Apply the same bounded-session pattern to the other YouTube transcript fetches under src/youtube_extension/backend, since they create the same unbounded client.

📍 Affects 4 files
  • src/agents/interactive_metadata_extractor.py#L106-L109 (this comment)
  • src/agents/process_video_with_mcp.py#L241-L246
  • src/agents/process_video_with_mcp.py#L257-L260
  • src/integration/youtube_api.py#L118-L120
  • src/mcp/mcp_video_processor.py#L713-L713
  • src/mcp/mcp_video_processor.py#L737-L737
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agents/interactive_metadata_extractor.py` around lines 106 - 109,
Introduce one shared transcript HTTP-client factory that returns a
requests.Session enforcing finite connect and read timeouts, then pass its
result via http_client= to every YouTubeTranscriptApi call. Update
src/agents/interactive_metadata_extractor.py lines 106-109,
src/agents/process_video_with_mcp.py lines 241-246 and 257-260,
src/integration/youtube_api.py lines 118-120, and src/mcp/mcp_video_processor.py
lines 713 and 737; apply the same bounded-session pattern to all transcript
fetches under src/youtube_extension/backend. Preserve existing proxy
configuration and fetch behavior while ensuring every transcript client uses the
shared timeout-enabled session.

Source: Path instructions

)

for i, entry in enumerate(transcript):
Expand Down
31 changes: 29 additions & 2 deletions src/agents/process_video_with_mcp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,25 @@
HAS_YTA = False
YouTubeTranscriptApi = None # type: ignore

try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError: # pragma: no cover - standalone execution outside the package
import sys as _sys
from pathlib import Path as _Path

# Running this module by path (the documented CLI entry point) leaves the
# repository's ``src`` directory off sys.path. Bootstrap it so the canonical
# helper resolves instead of silently disabling the proxy.
_sys.path.insert(0, str(_Path(__file__).resolve().parents[1]))
try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError as _exc: # pragma: no cover - helper genuinely unreachable
raise ImportError(
"youtube_extension.utils.proxy is required so transcript requests "
"honour WEBSHARE_PROXY_URL; refusing to continue with unproxied "
"egress."
) from _exc

try:
import yt_dlp # type: ignore
except Exception: # Provide a minimal stub so tests can patch attribute
Expand Down Expand Up @@ -219,7 +238,12 @@ async def _extract_transcript_with_rotation(self, video_id: str) -> list[dict[st
if YouTubeTranscriptApi is not None:
# youtube-transcript-api >=1.0 instance API
transcript = await loop.run_in_executor(
None, lambda: YouTubeTranscriptApi().fetch(video_id).to_raw_data()
None,
lambda: YouTubeTranscriptApi(
proxy_config=get_transcript_proxy_config()
)
.fetch(video_id)
.to_raw_data(),
)
if transcript:
return transcript
Expand All @@ -230,7 +254,10 @@ async def _extract_transcript_with_rotation(self, video_id: str) -> list[dict[st
try:
if YouTubeTranscriptApi is not None:
transcript_list = await loop.run_in_executor(
None, lambda: YouTubeTranscriptApi().list(video_id) # type: ignore[union-attr]
None,
lambda: YouTubeTranscriptApi( # type: ignore[union-attr]
proxy_config=get_transcript_proxy_config()
).list(video_id),
)
fetch_tasks = [
loop.run_in_executor(None, lambda t=t: t.fetch().to_raw_data())
Expand Down
23 changes: 22 additions & 1 deletion src/integration/youtube_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,25 @@
import httpx
from youtube_transcript_api import YouTubeTranscriptApi

try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError: # pragma: no cover - standalone execution outside the package
import sys as _sys
from pathlib import Path as _Path

# Running this module by path (the documented CLI entry point) leaves the
# repository's ``src`` directory off sys.path. Bootstrap it so the canonical
# helper resolves instead of silently disabling the proxy.
_sys.path.insert(0, str(_Path(__file__).resolve().parents[1]))
try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError as _exc: # pragma: no cover - helper genuinely unreachable
raise ImportError(
"youtube_extension.utils.proxy is required so transcript requests "
"honour WEBSHARE_PROXY_URL; refusing to continue with unproxied "
"egress."
) from _exc


@dataclass
class VideoMetadata:
Expand Down Expand Up @@ -96,7 +115,9 @@ async def get_transcript(
loop = asyncio.get_event_loop()
transcript = await loop.run_in_executor(
None,
lambda: YouTubeTranscriptApi().fetch(video_id, languages=languages).to_raw_data()
lambda: YouTubeTranscriptApi(proxy_config=get_transcript_proxy_config())
.fetch(video_id, languages=languages)
.to_raw_data()
)

return [
Expand Down
23 changes: 21 additions & 2 deletions src/mcp/mcp_video_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,25 @@ class IpBlocked(Exception):
except Exception:
HAS_YT_PROXY = False

try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError: # pragma: no cover - standalone execution outside the package
import sys as _sys
from pathlib import Path as _Path

# Running this module by path (the documented CLI entry point) leaves the
# repository's ``src`` directory off sys.path. Bootstrap it so the canonical
# helper resolves instead of silently disabling the proxy.
_sys.path.insert(0, str(_Path(__file__).resolve().parents[1]))
try:
from youtube_extension.utils.proxy import get_transcript_proxy_config
except ImportError as _exc: # pragma: no cover - helper genuinely unreachable
raise ImportError(
"youtube_extension.utils.proxy is required so transcript requests "
"honour WEBSHARE_PROXY_URL; refusing to continue with unproxied "
"egress."
) from _exc

# Configure logging
logging.basicConfig(
level=logging.INFO,
Expand Down Expand Up @@ -691,7 +710,7 @@ async def _direct_extraction():
# executor — otherwise it stalls the event loop and defeats the
# @timeout_protection / circuit-breaker hanging protection.
loop = asyncio.get_event_loop()
yt_api = YouTubeTranscriptApi()
yt_api = YouTubeTranscriptApi(proxy_config=get_transcript_proxy_config())
transcript = await loop.run_in_executor(
None,
lambda: yt_api.fetch(
Expand All @@ -715,7 +734,7 @@ async def _routed_extraction():
# These are blocking network calls — run them in an executor to keep
# the event loop free and let the timeout protection work.
loop = asyncio.get_event_loop()
yt_api = YouTubeTranscriptApi()
yt_api = YouTubeTranscriptApi(proxy_config=get_transcript_proxy_config())
transcript_list = await loop.run_in_executor(
None, lambda: yt_api.list(video_id)
)
Expand Down
Loading
Loading