fix(security): redact all proxy credentials and drop the phantom ecdsa dependency - #1438
Conversation
redact_proxy_credentials bailed out unless the configured WEBSHARE_PROXY_URL
appeared in the text byte-for-byte:
if not url or url not in text:
return text
It is called from exception handlers that log subprocess and HTTP failures, so
anything it skipped was written to logs verbatim. Reproduced leaks, all with
WEBSHARE_PROXY_URL=http://user:s3cr3t@proxy.internal:8080:
* host case-normalised by requests/urllib3 when re-rendering the URL
connect to http://user:s3cr3t@PROXY.INTERNAL:8080 -> leaked
* percent-encoded variant echoed back by yt-dlp
http://user:s3cr3t%40x@proxy.internal:8080 -> leaked
* a different proxy variable entirely, never equal to the configured one
HTTPS_PROXY=http://bob:hunter2@corp.proxy:3128 -> leaked
* a CalledProcessError repr of the argv
['yt-dlp','--proxy','http://u:p4ss@h:1'] -> leaked
Adds a second, generic pass: a scheme://user[:password]@ sweep that redacts
credentials regardless of which variable they came from or how they were
rendered. The exact-match pass is kept and runs first, because it preserves the
host so operators can still tell which proxy was in play.
The userinfo classes exclude the authority delimiters (whitespace, "/", "?",
"#") so a path or query containing "@" is never mistaken for credentials and a
match cannot span two URLs, but they permit a literal "@". RFC 3986 requires
"@" in userinfo to be percent-encoded while real proxy values carry a raw one;
because the classes are greedy the engine settles on the LAST "@" in the
authority, so http://user:pa@ss@host is consumed whole instead of the match
stopping at the first separator and leaving "ss@host" behind (#1113).
The helper now also accepts non-str input and never raises -- it is called from
except blocks, where raising would mask the original error. A hostile __str__
yields <unprintable error>; a redaction failure yields <redaction failed>.
Both fail closed rather than returning text that cannot be vouched for.
Applied to the canonical helper and to the drifted standalone copy in
shared/libs/ that the importlib fallback path uses, whose three logger calls in
the retry handler are the actual leak site.
Adds tests/unit/test_proxy_utils.py -- the module had no test coverage at all.
It parametrises over both implementations so neither can drift into leaking
alone, and covers over-redaction (an "@" in a path, query, or fragment, and a
bare host:port) since destroying diagnostics is its own failure.
Verified non-vacuous: 22 of the 38 new tests fail against the implementation on
main and all 38 pass after this change. Full unit suite 8035 passed, 0 failed.
No new ruff findings.
Re-cut from PR #1118, which was orphaned by the secret-purge force-push and
shares no ancestry with main. Tracked in #1378.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YcHjCZ6pGn6A5BeeoZ6eZi
…sory
python-jose was declared in pyproject.toml and requirements.txt but is never
imported by this codebase. The only occurrence is inside a string template in
backend/code_generator.py:
auth_imports = '''
from jose import JWTError, jwt
from passlib.context import CryptContext'''
That text is written into projects the generator emits, and the generator
writes those projects their own requirements.txt pinning python-jose (line
468). Parsing the file confirms it: zero jose imports at AST level anywhere in
src/, shared/, or scripts/.
Declaring it pulled in ecdsa, whose GHSA-wj6h-64fc-37mp has no patched release
-- so the advisory could not be resolved by upgrading, only by removing the
path to it. python-jose is ecdsa's sole dependent in the resolution, so
dropping it removes the advisory outright.
uv lock removes three packages: ecdsa, python-jose, and rsa. rsa goes because
python-jose was its only dependent too -- google-auth in this resolution
depends on cryptography and pyasn1-modules, not rsa. Verified zero residual
references to all three in uv.lock and zero AST-level imports of any of them.
Generated projects are unaffected: they install from the requirements.txt the
generator writes for them, which still pins python-jose[cryptography]==3.3.0.
Left in place: passlib is the same template-only case and could be dropped on
the same reasoning, but it carries no advisory and removing it is not needed
here. Noted in both manifests rather than changed unilaterally.
Verified: backend imports and serves 11 routes; code_generator (which holds the
template) imports; full unit suite 8035 passed, 0 failed.
Re-cut from PR #1156, which was orphaned by the secret-purge force-push and
shares no ancestry with main. Tracked in #1378.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YcHjCZ6pGn6A5BeeoZ6eZi
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 PR Validation |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. OpenSSF Scorecard
Scanned Files
|
CI caught what my local run missed: three tests in test_code_generator.py failed with ModuleNotFoundError: No module named 'jose'. TestGeneratedFastAPIBehaviour does not inspect the generated source, it *executes* the FastAPI app that code_generator emits -- deliberate regression cover for #1257, where the template shipped placeholder endpoints returning 200 so a generated project passed a naive smoke test while being non-functional. The emitted app imports jose, so removing the dependency broke those tests. I checked src/, shared/ and scripts/ for jose imports and found none, but never checked tests/. My local suite passed only because the venv still had python-jose installed from an earlier editable install; CI installs fresh. Moving python-jose to the dev extra would have fixed CI while leaving ecdsa in uv.lock, so the advisory would have survived -- and every generated project would still inherit it. Instead the template now emits PyJWT: -from jose import JWTError, jwt +import jwt +from jwt import PyJWTError encode/decode signatures are identical; only the exception type changes. PyJWT is maintained and depends on nothing with an open advisory, whereas python-jose's ecdsa (GHSA-wj6h-64fc-37mp) has no patched release. Generated projects get pyjwt>=2.10.1 in their requirements.txt instead of python-jose[cryptography]==3.3.0. This changes generator output -- called out explicitly on the PR so it can be objected to -- but a generated project should not ship a known-vulnerable transitive dependency. pyjwt is added to the dev extra, not the runtime dependencies: EventRelay never imports it, only the generated app the tests execute does. Verified in a venv with jose uninstalled, matching CI: 96 code_generator tests pass, full unit suite 8035 passed, 0 failed. ecdsa, python-jose and rsa all absent from uv.lock; pyjwt present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YcHjCZ6pGn6A5BeeoZ6eZi
CI caught a real miss, and the fix changes generator output — please read
What I got wrongI verified "zero My local suite passed 8,035/0 only because the venv still had Those tests are regression cover for #1257, where the template shipped placeholder endpoints returning 200 so a generated project passed a naive smoke test while being non-functional. Making them skip when Why not just move it to the dev extraThat fixes CI while leaving What I did insteadThe template now emits PyJWT: -from jose import JWTError, jwt
+import jwt
+from jwt import PyJWTError
This changes generator output. Projects generated with the
Verification, this time in an environment matching CII uninstalled
Head is now Generated by Claude Code |
Canonical issue
Closes #1437
Outcome
Lands the two security fixes from #1378 that were orphaned by the secret-purge force-push. Both original PRs (#1118, #1156) share no ancestry with
main, so neither could be rebased or cherry-picked — both were reimplemented against currentmain, and in one case the original premise had to be re-verified before it could be trusted.Proxy credentials stop leaking into logs.
redact_proxy_credentialsbailed out unless the configuredWEBSHARE_PROXY_URLappeared byte-for-byte, so a host that urllib3 had case-normalised, a percent-encoded form echoed by yt-dlp, aCalledProcessErrorargv repr, or any other proxy variable passed straight through into log output.The
ecdsaadvisory is cleared by removing the path to it. GHSA-wj6h-64fc-37mp has no patched release, so upgrading was never an option.python-josewas its sole dependent and is never imported.Scope
src/youtube_extension/utils/proxy.py— two-pass redaction; accepts non-str; never raises.shared/libs/youtube_proxy.py— the drifted standalone copy, whose three logger calls in the retry handler are the actual leak site.tests/unit/test_proxy_utils.py— new; the module had no coverage at all.pyproject.toml,requirements.txt,uv.lock— droppython-jose.passlib— the same template-only phantom, but no advisory. Flagged in both manifests, not removed unilaterally.Risk
@in a path, query, or fragment, and a barehost:port, all pass through untouched, and a match can never span two URLs. The helper is also now total: a hostile__str__yields<unprintable error>, a redaction failure yields<redaction failed>. Both fail closed rather than raising inside anexceptblock or returning text that cannot be vouched for.uv lockalso removedrsa; verified its only dependent waspython-josetoo, and thatgoogle-authin this resolution needscryptography+pyasn1-modules, notrsa. Zero residual references toecdsa,rsa, orpython-joseinuv.lock, and zero AST-level imports of any of them.git revert. Two independent commits; either can be reverted alone.Verification
Head
de6f36cae.main, all 38 pass after the change. Checked by restoringmain's two source files and re-running.mainand confirmed clean after.shared/libsimplementations, so neither can drift into leaking alone. Regex source verified byte-identical across the two.python-josereally is phantom — verified by AST parse, not grep: zerojoseimports insrc/,shared/,scripts/. The one textual hit is inside a string template.code_generator.pywrites its ownrequirements.txtpinningpython-jose[cryptography]==3.3.0for the project it emits.code_generator(which holds the template) imports.shared/libs/youtube_proxy.pyholds at 8 pre-existing, unchanged.Production evidence
Not applicable — no runtime behaviour changes for users. The redaction change only affects what is written to logs (strictly less credential material), and the dependency change removes packages that were never imported. Under
MERGE_POLICY.mdgate 4, previews gateapps/web/**; this touches none.A correction worth recording
The original #1156 was titled "drop phantom python-jose". On current
mainthat premise no longer looked true —grepfindsfrom jose import JWTError, jwtincode_generator.py. Had I trusted the old PR's framing, I would have either skipped the fix as stale or removed a dependency that appeared to be in use.Parsing the file settled it: the hit is inside a string literal the generator emits, and there are zero
joseimports at AST level. The premise held; the grep was just the wrong instrument. Same shape as themerge-baselesson from the branch audit — the cheap signal disagreed with the true one.Agent handoff
passlibGenerated by Claude Code