Skip to content

fix(espn): send an identifying User-Agent so ESPN stops returning 403 - #254

Merged
ChuckBuilds merged 3 commits into
mainfrom
claude/espn-api-outage-od3208
Aug 5, 2026
Merged

fix(espn): send an identifying User-Agent so ESPN stops returning 403#254
ChuckBuilds merged 3 commits into
mainfrom
claude/espn-api-outage-od3208

Conversation

@ChuckBuilds

@ChuckBuilds ChuckBuilds commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Around 11:00 EDT on 2026-08-04, ESPN's site.api began rejecting the User-Agent strings these plugins send, and every ESPN-backed scoreboard started returning 403 Client Error: Forbidden. A device that had been running fine logged 287 ESPN errors in a day.

The filter is the opposite of the usual one

Probed site.api across agents, using requests as the plugins do:

User-Agent result
bare LEDMatrix/1.0 403 (with or without Accept)
browser string (Chrome/Safari) 403 — no header rescues it
LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix) 200
requests / urllib / curl defaults 200

ESPN rejects browser-style strings outright and bare custom tokens, while accepting honest client tokens or an agent that identifies the client and links to it. The reflex fix — "send a browser User-Agent" — is now the one change guaranteed to stay blocked.

Confirmed independently: ha-teamtracker#355 hit the same wall and switched from a browser agent to curl/8.20.0 (merged 2026-08-05).

The change

Rather than guess at the blast radius, every User-Agent literal in the repo was collected and probed. Ten blocked strings, across 24 files in 14 plugins:

  • the LEDMatrix/1.0 family in the data_sources.py / dynamic_team_resolver.py files (16 files)
  • per-plugin tokens: LEDMatrix-F1/1.0, LEDMatrix Masters Plugin/2.0 and /2.1, LEDMatrix Baseball Plugin/1.0, LEDMatrix/2.0, LEDMatrix-NewsPlugin/1.0 (RSS Reader)
  • browser strings in nfl-draft/manager.py and masters-tournament/download_assets.py

All now send one agent carrying the project URL — a single string, so the next ESPN change is one grep rather than ten. Re-running the sweep afterwards reports zero blocked agents on any ESPN caller.

Callers that reach other services are deliberately untouched: ledmatrix-flights, ledmatrix-stocks and stock-news send a browser agent to hosts ESPN's change never involved.

The probe was reporting healthy through the outage

scripts/check_espn_api.py (added earlier on this branch) printed "All 10 endpoints healthy under every User-Agent. ESPN is not the problem" while 7 of 10 endpoints were visibly refusing an agent. It only tested for the old shape — browser works, ours does not — so an inverted filter registered as fine, and the remedy it printed, "send a browser User-Agent from every ESPN caller", was precisely the change that would have kept everything broken.

Fixed: the verdict is direction-agnostic, names which agents were accepted and which refused instead of assuming, and splits profiles into what the plugins actually ship versus controls kept only to characterise the filter. A split that leaves every shipped agent working is now a warning rather than a failure, so the non-zero exit still means "act now".

Verification

On a live 512x64 device, across a service restart:

  • ESPN errors: 287 in the preceding day → 0
  • live MLB games fetching again (mlb_live=True, MLB=4 live)
  • standings and game summaries no longer 403

Notes for review

🤖 Generated with Claude Code

https://claude.ai/code/session_01Udr6MfaFLUPhX5Fgo67Jf5

Summary by CodeRabbit

  • Bug Fixes

    • Improved ESPN data and asset requests across sports plugins by providing consistent project identification, helping prevent access failures.
    • Updated plugin versions and release metadata for the affected scoreboard, tournament, news, and draft plugins.
  • New Features

    • Added an ESPN API diagnostic tool with endpoint checks, response validation, connectivity reporting, and human-readable or JSON output.
  • Chores

    • Refreshed catalog metadata for 14 plugins.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The change standardizes identifying ESPN request headers across plugins, adds an ESPN API diagnostic command, and updates plugin versions, release notes, and catalog metadata.

Changes

ESPN request identification

Layer / File(s) Summary
ESPN API diagnostic tool
scripts/check_espn_api.py
Adds endpoint probing, User-Agent comparison, response validation, report rendering, and diagnostic exit statuses.
Identifying User-Agent headers
plugins/*/{data_sources.py,dynamic_team_resolver.py,manager.py,...}
Updates ESPN, RSS, headshot, and related HTTP requests to include the LEDMatrix project URL.
Plugin release metadata
plugins.json, plugins/*/manifest.json
Updates plugin versions, release histories, and metadata dates for the affected plugins.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant Diagnostic
  participant ESPN
  participant Report
  CLI->>Diagnostic: Run check_espn_api.py
  Diagnostic->>ESPN: Probe endpoints with request profiles
  ESPN-->>Diagnostic: Return response or connection error
  Diagnostic->>Report: Validate response keys and classify results
  Report-->>CLI: Print report and return status
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 77.42% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: adding an identifying User-Agent to prevent ESPN 403 responses.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/espn-api-outage-od3208

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codacy-production

codacy-production Bot commented Aug 5, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 73 complexity

Metric Results
Complexity 73

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@ChuckBuilds ChuckBuilds changed the title feat(scripts): add an ESPN endpoint probe that names the failure fix(espn): send an identifying User-Agent so ESPN stops returning 403 Aug 5, 2026
claude added 3 commits August 5, 2026 16:49
"The ESPN plugins went blank" arrives as a symptom and rarely as a
diagnosis. The plugins call ESPN's undocumented API from ~15 places and,
critically, do not speak with one voice: some callers send LEDMatrix/1.0,
many send nothing at all and get the stdlib/requests default, and a few
already send a browser string. Anti-bot filtering discriminates on exactly
that header, so the same outage can hit some plugins and spare others,
which makes the reports hard to read.

check_espn_api.py probes each endpoint family under all three User-Agent
profiles and turns the result into a diagnosis: a browser-works/ours-fail
split means header filtering and a header fix; failure under every agent
means the URL moved or the response shape changed.

It also separates "never got an HTTP response" from "ESPN returned an
error". A denied proxy or a DNS failure otherwise looks identical to a
withdrawn endpoint, and sends whoever runs it off rewriting URLs that were
never broken. That case now exits 2 and says so.

Endpoint checks assert an expected key rather than a bare 200, since ESPN
can serve a cheerful 200 whose body no longer holds the field the renderer
reads. An empty events list stays a note, not a failure, so it does not cry
wolf on an off-day.

No plugin code or versions touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017mSAZ7o55QKTP8KQm3cv5c
Codacy flagged the urlopen call (bandit B310, medium): urlopen honours
file:// and custom schemes, so a URL reaching it unchecked is a path
traversal waiting for a careless edit. The endpoints here are module
constants today, but probe() is generic and the next person to add a row
gets the guard for free.

Follows the convention already set in check_team_pickers.py, with one
difference: probe() documents that it never raises, so a bad scheme comes
back as an ordinary failure result rather than an exception that would
abort the remaining probes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017mSAZ7o55QKTP8KQm3cv5c
Around 11:00 EDT on 2026-08-04 ESPN's site.api began rejecting the agents
these plugins send, and every ESPN-backed scoreboard started returning
`403 Client Error: Forbidden`. A device that had been running fine logged
287 ESPN errors in a day.

The filter is not the familiar one. Probing site.api across agents, using
requests as the plugins do:

    bare 'LEDMatrix/1.0'                      403   (with or without Accept)
    browser string                            403   (no header rescues it)
    'LEDMatrix/1.0 (+https://github.com/...)'  200
    requests / urllib / curl defaults         200

ESPN rejects browser-style strings outright and bare custom tokens, and
accepts honest client tokens or an agent that identifies the client and
links to it. The instinct to "just send a browser User-Agent" is now
exactly backwards — that is the one thing guaranteed to stay blocked.

Sweeping every User-Agent literal in the repo and probing each turned up
ten blocked strings across 24 files in 14 plugins: the `LEDMatrix/1.0`
family in the data_sources/dynamic_team_resolver files, per-plugin tokens
like `LEDMatrix-F1/1.0` and `LEDMatrix Masters Plugin/2.1`, and browser
strings in nfl-draft and masters-tournament. All now send one agent
carrying the project URL — one string, so the next ESPN change is one
grep rather than ten.

Callers that reach other services are deliberately untouched:
ledmatrix-flights, ledmatrix-stocks and stock-news send a browser agent
to hosts ESPN's change never involved.

Also fixes the probe this branch added, which reported "ESPN is not the
problem" throughout the outage. It only tested for the old shape —
browser works, ours does not — so an inverted filter read as healthy, and
the fix it printed (send a browser agent) was the change that would have
kept everything broken. The verdict is direction-agnostic now, names
which agents were accepted and refused rather than assuming, and
separates what the plugins ship from controls kept to characterise the
filter.

basketball, hockey and lacrosse skip a patch number so this cannot
collide with the versions PR #252 already claims.

Verified on a live device: ESPN errors went from a steady stream to zero
across a restart, with live MLB games fetching again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Udr6MfaFLUPhX5Fgo67Jf5
@ChuckBuilds
ChuckBuilds force-pushed the claude/espn-api-outage-od3208 branch from 41532b1 to f8251fd Compare August 5, 2026 20:50
@ChuckBuilds
ChuckBuilds merged commit 2c35486 into main Aug 5, 2026
3 of 4 checks passed

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
scripts/check_espn_api.py (1)

128-132: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Match the shipped request-header profiles.

shipped sends Accept: application/json, but plugins/march-madness/manager.py:146, plugins/nfl-draft/manager.py:281, and plugins/masters-tournament/masters_data.py:43 send only the User-Agent. Add separate profiles that match actual shipped requests and urllib defaults, or remove the explicit Accept header where callers do not send it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/check_espn_api.py` around lines 128 - 132, Update the shipped
request-header profile definitions in the ESPN API checker so they match the
actual callers: remove the explicit Accept header from the profile used by
requests that send only User-Agent, or add distinct profiles for callers with
different requests/urllib defaults. Ensure each profile accurately reflects the
headers sent by the referenced plugin managers and masters data loader.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/basketball-scoreboard/data_sources.py`:
- Line 49: Scope the shared User-Agent to ESPN requests only: in
plugins/basketball-scoreboard/data_sources.py at lines 49-49, update
SoccerAPIDataSource.get_headers() and MLBAPIDataSource usage so non-ESPN
requests do not inherit it; in plugins/f1-scoreboard/f1_data.py at lines 55-55,
update _fetch_json() to apply the header only for ESPN calls or use separate
service-specific sessions, preserving appropriate headers for Jolpi and OpenF1.

In `@plugins/news/manager.py`:
- Line 925: In plugins/news/manager.py:925-925, update the request-header
selection to use the ESPN-specific User-Agent only for ESPN feed hosts, while
preserving the previous header for Google News, Covering the Corner, and custom
feeds. In plugins/nfl-draft/manager.py:377-377, restore the prior Tankathon
request header unless Tankathon is explicitly intended to be included in the
supported ESPN-scoped change.

In `@scripts/check_espn_api.py`:
- Around line 156-172: Update probe to use a custom urllib redirect handler that
validates each redirect target with is_safe_redirect_url(...) and rejects any
non-HTTPS destination before following it. Preserve the existing initial URL
scheme check, request headers, timeout, and never-raises failure behavior while
ensuring the handler is used by the urlopen call.

---

Nitpick comments:
In `@scripts/check_espn_api.py`:
- Around line 128-132: Update the shipped request-header profile definitions in
the ESPN API checker so they match the actual callers: remove the explicit
Accept header from the profile used by requests that send only User-Agent, or
add distinct profiles for callers with different requests/urllib defaults.
Ensure each profile accurately reflects the headers sent by the referenced
plugin managers and masters data loader.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9b62aeb8-d813-4ed0-baa9-65f3e992dbb6

📥 Commits

Reviewing files that changed from the base of the PR and between fb8459d and f8251fd.

📒 Files selected for processing (40)
  • plugins.json
  • plugins/afl-scoreboard/data_sources.py
  • plugins/afl-scoreboard/manifest.json
  • plugins/baseball-scoreboard/data_sources.py
  • plugins/baseball-scoreboard/dynamic_team_resolver.py
  • plugins/baseball-scoreboard/logo_manager.py
  • plugins/baseball-scoreboard/manifest.json
  • plugins/basketball-scoreboard/data_sources.py
  • plugins/basketball-scoreboard/dynamic_team_resolver.py
  • plugins/basketball-scoreboard/manifest.json
  • plugins/f1-scoreboard/f1_data.py
  • plugins/f1-scoreboard/manifest.json
  • plugins/football-scoreboard/data_sources.py
  • plugins/football-scoreboard/dynamic_team_resolver.py
  • plugins/football-scoreboard/manifest.json
  • plugins/hockey-scoreboard/data_sources.py
  • plugins/hockey-scoreboard/dynamic_team_resolver.py
  • plugins/hockey-scoreboard/manifest.json
  • plugins/lacrosse-scoreboard/data_sources.py
  • plugins/lacrosse-scoreboard/dynamic_team_resolver.py
  • plugins/lacrosse-scoreboard/manifest.json
  • plugins/lacrosse-scoreboard/test_lacrosse_plugin.py
  • plugins/march-madness/manager.py
  • plugins/march-madness/manifest.json
  • plugins/masters-tournament/download_assets.py
  • plugins/masters-tournament/logo_loader.py
  • plugins/masters-tournament/manifest.json
  • plugins/masters-tournament/masters_data.py
  • plugins/news/manager.py
  • plugins/news/manifest.json
  • plugins/nfl-draft/manager.py
  • plugins/nfl-draft/manifest.json
  • plugins/nrl-scoreboard/data_sources.py
  • plugins/nrl-scoreboard/manifest.json
  • plugins/soccer-scoreboard/data_sources.py
  • plugins/soccer-scoreboard/manifest.json
  • plugins/ufc-scoreboard/data_sources.py
  • plugins/ufc-scoreboard/headshot_downloader.py
  • plugins/ufc-scoreboard/manifest.json
  • scripts/check_espn_api.py

"""Get headers for API requests."""
return {
'User-Agent': 'LEDMatrix/1.0',
'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files:"
git ls-files | rg '(^|/)plugins/(basketball-scoreboard|f1-scoreboard)/(data_sources|f1_data)\.py$' || true

echo
echo "basketball data_sources outline:"
ast-grep outline plugins/basketball-scoreboard/data_sources.py --view expanded || true

echo
echo "basketball relevant lines:"
cat -n plugins/basketball-scoreboard/data_sources.py | sed -n '1,160p'

echo
echo "f1 f1_data outline:"
ast-grep outline plugins/f1-scoreboard/f1_data.py --view expanded || true

echo
echo "f1 relevant lines:"
cat -n plugins/f1-scoreboard/f1_data.py | sed -n '1,180p'

Repository: ChuckBuilds/ledmatrix-plugins

Length of output: 19333


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "soccer class and call sites:"
cat -n plugins/basketball-scoreboard/data_sources.py | sed -n '160,340p'

echo
echo "F1 requests using _fetch_json / bases:"
python3 - <<'PY'
from pathlib import Path
p = Path('plugins/f1-scoreboard/f1_data.py')
s = p.read_text()
bases = ['ESPN_BASE', 'JOLPI_BASE', 'OPENF1_BASE']
for i,line in enumerate(s.splitlines(),1):
    if '_fetch_json(' in line or any(b in line for b in bases):
        print(f'{i}: {line}')
PY

echo
echo "rest of F1 relevant _fetch_json usage:"
cat -n plugins/f1-scoreboard/f1_data.py | sed -n '108,90p' | sed -n '1,80p'
cat -n plugins/f1-scoreboard/f1_data.py | sed -n '138,750p' | grep -n "_fetch_json\|ESPN_BASE\|JOLPI_BASE\|OPENF1_BASE" || true

Repository: ChuckBuilds/ledmatrix-plugins

Length of output: 10109


Scope the new User-Agent per service.

Both locations send the shared User-Agent beyond ESPN calls.

  • plugins/basketball-scoreboard/data_sources.py: SoccerAPIDataSource.get_headers() calls super().get_headers() and also uses MLBAPIDataSource, so keep this value only for ESPN requests or override it in the non-ESPN classes.
  • plugins/f1-scoreboard/f1_data.py: _fetch_json() is used by Jolpi and OpenF1 callers too, so override headers only for ESPN or use separate service sessions.
📍 Affects 2 files
  • plugins/basketball-scoreboard/data_sources.py#L49-L49 (this comment)
  • plugins/f1-scoreboard/f1_data.py#L55-L55
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/basketball-scoreboard/data_sources.py` at line 49, Scope the shared
User-Agent to ESPN requests only: in
plugins/basketball-scoreboard/data_sources.py at lines 49-49, update
SoccerAPIDataSource.get_headers() and MLBAPIDataSource usage so non-ESPN
requests do not inherit it; in plugins/f1-scoreboard/f1_data.py at lines 55-55,
update _fetch_json() to apply the header only for ESPN calls or use separate
service-specific sessions, preserving appropriate headers for Jolpi and OpenF1.

Comment thread plugins/news/manager.py
self.logger.info(f"Fetching headlines from {feed_name}...")
headers = {
'User-Agent': 'LEDMatrix-NewsPlugin/1.0 (RSS Reader)'
'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep non-ESPN User-Agent values unchanged.

The PR objective limits this change to ESPN callers. These requests target other services and now receive the ESPN-specific header.

  • plugins/news/manager.py#L925-L925: select the identifying header only for ESPN feed hosts. Preserve the prior header for Google News, Covering the Corner, and custom feeds.
  • plugins/nfl-draft/manager.py#L377-L377: restore the prior Tankathon request header, or explicitly include Tankathon in the supported change scope.
📍 Affects 2 files
  • plugins/news/manager.py#L925-L925 (this comment)
  • plugins/nfl-draft/manager.py#L377-L377
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/news/manager.py` at line 925, In plugins/news/manager.py:925-925,
update the request-header selection to use the ESPN-specific User-Agent only for
ESPN feed hosts, while preserving the previous header for Google News, Covering
the Corner, and custom feeds. In plugins/nfl-draft/manager.py:377-377, restore
the prior Tankathon request header unless Tankathon is explicitly intended to be
included in the supported ESPN-scoped change.

Comment thread scripts/check_espn_api.py
Comment on lines +156 to +172
def probe(url, headers, timeout):
"""Fetch url and return a result dict. Never raises."""
# The URLs come from the table above, but pin the scheme rather than trusting
# them: urlopen would honour file:// or a custom scheme if an entry ever
# arrived from somewhere less trustworthy. Reported as an ordinary failure so
# the promise above holds and one bad row cannot abort the whole run.
if not url.startswith("https://"):
return {"ok": False, "status": None, "error": f"refusing non-HTTPS URL: {url!r}"}

request = urllib.request.Request(url)
for name, value in headers.items():
request.add_header(name, value)

try:
# B310 is a syntactic blacklist rule and fires on the call regardless of
# the scheme guard above, which is what actually makes this safe.
with urllib.request.urlopen(request, timeout=timeout) as response: # nosec B310

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== git status/stat =="
git status --short
git diff --stat || true

echo "== file outline/size =="
wc -l scripts/check_espn_api.py
ast-grep outline scripts/check_espn_api.py --view compact || true

echo "== relevant source =="
sed -n '120,190p' scripts/check_espn_api.py

echo "== full script symbols/usages of probe/urllib =="
rg -n "def probe|urlopen|urllib\.request|redirect|https://|http://" scripts/check_espn_api.py

echo "== Python urllib redirect behavior probe if available =="
python3 - <<'PY'
import urllib.request
import tempfile, os, http.server, socketserver, threading, time
port = 0
srv = None

class RedirectHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/redirect":
            self.send_response(302)
            self.send_header("Location", "http://httpbin.invalid/http-redirect-host")
            self.end_headers()
        else:
            # No handler for redirected GET; urllib should make a GET and then fail.
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"origin")

with tempfile.TemporaryDirectory() as d:
    path = os.path.join(d, "simple.html")
    with open(path, "wb") as f:
        f.write(b"<html></html>")
    # urllib follows FileRedirectHandler in Python standard library for file://;
    # show that non-HTTPS initial URL guard can be bypassed if caller did not check scheme.
    req = urllib.request.Request("file://" + path)
    try:
        with urllib.request.urlopen(req, timeout=2) as r:
            print("file_protocol:", r.url, r.status)
    except Exception as e:
        print("file_exception:", type(e).__name__, str(e))

srv = http.server.HTTPServer(("127.0.0.1", port), RedirectHandler)
port = srv.server_address[1]
t = threading.Thread(target=srv.serve_forever, daemon=True)
t.start()
time.sleep(0.2)
try:
    req = urllib.request.Request(f"http://127.0.0.1:{port}/redirect")
    with urllib.request.urlopen(req, timeout=2) as r:
        print("protocol:", r.url, r.status)
except Exception as e:
    print("redirect_exception:", type(e).__name__, str(e), getattr(e, "reason", None))
finally:
    srv.shutdown()
    srv.server_close()
PY

Repository: ChuckBuilds/ledmatrix-plugins

Length of output: 5993


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== git status/stat =="
git status --short || true
git diff --stat || true

echo "== file outline/size =="
wc -l scripts/check_espn_api.py || true
ast-grep outline scripts/check_espn_api.py --view compact || true

echo "== relevant source =="
sed -n '120,190p' scripts/check_espn_api.py || true

echo "== full script symbols/usages of probe/urllib =="
rg -n "def probe|urlopen|urllib\.request|redirect|https://|http://" scripts/check_espn_api.py || true

echo "== Python urllib redirect behavior probe if available =="
python3 - <<'PY'
import urllib.request
import tempfile, os, http.server, socketserver, threading, time

class RedirectHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/redirect":
            self.send_response(302)
            self.send_header("Location", "http://httpbin.invalid/http-redirect-host")
            self.end_headers()
        else:
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"origin")

with tempfile.TemporaryDirectory() as d:
    path = os.path.join(d, "simple.html")
    with open(path, "wb") as f:
        f.write(b"<html></html>")
    req = urllib.request.Request("file://" + path)
    try:
        with urllib.request.urlopen(req, timeout=2) as r:
            print("file_protocol:", r.url, r.status)
    except Exception as e:
        print("file_exception:", type(e).__name__, str(e))

srv = http.server.HTTPServer(("127.0.0.1", 0), RedirectHandler)
port = srv.server_address[1]
t = threading.Thread(target=srv.serve_forever, daemon=True)
t.start()
time.sleep(0.2)
try:
    req = urllib.request.Request(f"http://127.0.0.1:{port}/redirect")
    with urllib.request.urlopen(req, timeout=2) as r:
        print("protocol:", r.url, r.status)
except Exception as e:
    print("redirect_exception:", type(e).__name__, str(e), getattr(e, "reason", None))
finally:
    srv.shutdown()
    srv.server_close()
PY

Repository: ChuckBuilds/ledmatrix-plugins

Length of output: 5954


Reject redirects to non-HTTPS destinations.

The scheme check only covers the initial https:// URL. urllib.request.urlopen follows redirects. If a redirect lands on http://, the probe still makes a non-HTTPS request. Use a redirect handler that calls is_safe_redirect_url(...) and rejects unsafe redirect targets before urlopen proceeds.

🧰 Tools
🪛 ast-grep (0.45.0)

[warning] 171-171: Request-controlled URL passed to urlopen; validate against an allowlist to prevent SSRF.
Context: urllib.request.urlopen(request, timeout=timeout)
Note: [CWE-918] Server-Side Request Forgery (SSRF).

(urlopen-unsanitized-data)

🪛 Ruff (0.16.1)

[error] 165-165: Audit URL open for permitted schemes. Allowing use of file: or custom schemes is often unexpected.

(S310)


[error] 172-172: Audit URL open for permitted schemes. Allowing use of file: or custom schemes is often unexpected.

(S310)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/check_espn_api.py` around lines 156 - 172, Update probe to use a
custom urllib redirect handler that validates each redirect target with
is_safe_redirect_url(...) and rejects any non-HTTPS destination before following
it. Preserve the existing initial URL scheme check, request headers, timeout,
and never-raises failure behavior while ensuring the handler is used by the
urlopen call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants