Make shuck run on Windows - #1
Open
EthanBGilbert wants to merge 1 commit into
Open
Conversation
Four portability fixes. Without the first one shuck hangs on Windows with no output at all, so this is the difference between "works" and "doesn't". - The Chrome profile dir was hardcoded to /tmp. On Windows os.path.join turns that into `/tmp\shuck-profile-N`, which Chrome rejects, exiting 21 before the DevTools port ever opens; shuck then sits in its 60-round poll loop and gives up with a misleading "is Chrome installed?". Use tempfile.gettempdir(), which is still /tmp (or $TMPDIR) on POSIX. - CHROME_CANDIDATES listed only macOS bundle paths, and the shutil.which() probes look for `google-chrome`/`chromium`, which never match chrome.exe and aren't on PATH on Windows anyway. That made SHUCK_CHROME mandatory. Add the standard Windows install locations for Chrome/Chromium/Brave/Edge. - Document names are routinely non-ASCII, and on a cp1252 console printing one raised UnicodeEncodeError *midway through the listing*, so the run exited 1 having printed only the links before the first accented name. A register with "Ngati Whatua assessment" (macrons) as its second entry reported 1 of 7 links. Pin stdout/stderr to UTF-8 up front. - The DevTools call to our own headless Chrome went through the system proxy. Windows reads proxy config from the registry, where the "<local>" bypass does not exempt an IP literal like 127.0.0.1, so /json/list got proxied; when the proxy is up but Chrome is not yet, the 502 it returns is indistinguishable from Chrome still booting. Bypass proxies for that one loopback call. Downloads still honour the proxy, as they should. Verified on Windows 11 / Python 3.12 / Chrome headless, against a fixture reproducing the three obstacles from the README (lazy-load-on-click accordions, details that open empty, "Download" anchor text): list, --match, --json, --get, --no-expand and --dom all behave, downloads land with their Unicode names intact, and the temp profile is cleaned up. macOS/Linux behaviour is unchanged: the added paths simply don't exist there, and gettempdir() resolves to the same /tmp as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found this via the README, tried it on Windows 11, and it hung with no output. Four fixes, in rough order of how badly they bite.
The fatal one: the Chrome profile path
profile = os.path.join("/tmp", f"shuck-profile-{port}")becomes/tmp\shuck-profile-54321on Windows. Chrome refuses that as a--user-data-dirand exits 21 immediately, so the DevTools port never opens, the 60-round poll loop spins for 30s, and you getshuck: could not reach Chrome DevTools — is Chrome installed and runnable?— which points at the wrong thing entirely, since Chrome is installed and did launch.tempfile.gettempdir()is still/tmp(or$TMPDIR) on POSIX, so this is a no-op there.CHROME_CANDIDATEShas no Windows entriesThe list is macOS bundle paths plus
shutil.which("google-chrome"|"chromium"|"chromium-browser"), none of which matchchrome.exe— and the browsers aren't onPATHon Windows regardless. SoSHUCK_CHROMEwas effectively mandatory. Added the standard install locations for Chrome/Chromium/Brave/Edge.UnicodeEncodeErrortruncates the listingThis one is nastier than a cosmetic glitch. On a cp1252 console, printing a document name containing a macron, accent or em dash raises mid-loop, so shuck exits 1 having printed only the links before the offending name. Against a register whose second document was
Ngāti Whātua assessment — Pūkaki, I got 1 link out of 7 and a traceback — a partial result that looks like a complete one if you're reading the first few lines.Pinned
stdout/stderrto UTF-8 witherrors="backslashreplace". This also fixes the✓/✗download markers, which were printing as literal✓/✗.The DevTools call goes through the system proxy
Windows reads proxy config from the registry, where the
<local>bypass does not exempt an IP literal like127.0.0.1(Python'sproxy_bypass_registryonly treats dotless hosts as local). So/json/listgets proxied. With a proxy running on127.0.0.1:8080I gotHTTPError 502back — indistinguishable from Chrome still booting, and it'd fail outright if the proxy were down.Bypassed proxies for that one loopback call only. Downloads still go through the proxy, which is the correct behaviour.
Testing
Windows 11, Python 3.12, headless Chrome. Built a fixture reproducing all three obstacles from the README — accordions that lazy-load on the click rather than the open state,
<details>that open empty, andDownload/Viewanchor text with the real name in a neighbouring cell:shuck: 7, with names correctly resolved from the sibling cells.--match,--json,--no-expand,--domall behave.--getwrites files with their Unicode names intact on NTFS.macOS/Linux behaviour should be unchanged: the added candidate paths don't exist there so
find_chrome()skips them,gettempdir()resolves to the same/tmp, and the stream reconfigure is a no-op on an already-UTF-8 console. I don't have a mac to hand to confirm, so worth a second pair of eyes on that.Happy to split this into separate commits if you'd rather take them piecemeal — the profile-path fix is the only one that's load-bearing.
🤖 Generated with Claude Code