Pr2142 core features v2 - #2191
Merged
Merged
Conversation
Includes throttling the progress update to every 200ms (instead of every read chunk, which could fire thousands of times/sec) and a fix for the throttled byte tracking undercounting the byte-based progress bar / dropping the final <200ms chunk of every download. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Detects ENOSPC locale-independently: IOException's message for ENOSPC is localized by the JVM, so a plain 'No space left on device' string match only works on English-locale systems. Falls back to checking the filesystem's usable space via FileStore when the message doesn't match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Generic IOExceptions now notify downloadErrored and give up instead of silently falling through the retry loop forever. Redirects and retriable 5xx responses use continue to retry directly rather than the old throw-to-retry trick, which the above change broke (5xx responses stopped retrying after a single attempt since the generic IOException handler now returns immediately instead of falling through the loop). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Introduces TokenedUrlGetter and RipUrlId: some hosts serve media via signed or time-limited URLs, so downloads need a way to fetch a fresh URL for the same logical item rather than reusing a possibly-expired one, and a stable identity (RipUrlId) to track/dedupe that item across albums independent of which URL was used to fetch it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Each ripper previously maintained its own separate DownloadThreadPool for resolving direct image URLs from intermediate "detail" pages (one per ripper subclass), in addition to the shared download pool -- now unified into a shared crawler pool and ripper pool per rip. Also fixes a real race: if all downloads on a page were skipped, the old itemsPending.isEmpty()-based completion check could fire before the crawler pool had finished discovering and queuing every item (e.g. a later page's items not yet queued), letting the next ripper in the queue start early. Completion is now tracked via itemsSeen vs. itemsTotal counters instead, with the crawler pool required to fully drain before the ripper pool's completion is even checked. Bundled fixes for three deadlocks the itemsSeen/itemsTotal approach introduced: - downloadCompleted/downloadErrored/downloadExists/downloadSkipped gated their itemsPending bookkeeping behind an observer-null check meant only for the GUI notification. Neither the CLI (App.java) nor the test harness ever set an observer, so itemsPending never emptied outside the GUI -- every non-GUI rip hung forever. - The crawler-pool wait polled for getScheduledThreadCount() to reach the crawl loop's item count, which is wrong for any ripper whose downloadURL() doesn't submit a crawler-pool thread per item (e.g. DribbbleRipper, which calls addURLToDownload() directly) -- the count never reached the target. Removed the wait: crawler threads are always submitted synchronously before this call, so there was nothing left to wait for scheduling-wise. - itemsSeen was only incremented past the URL-history "already downloaded, skip" check, deep in the call chain. In test mode this never mattered (only one item is ever processed), but in real CLI usage -- re-ripping an album where some or all items were already downloaded -- every skipped item bypassed the increment, so itemsSeen could never reach itemsTotal. Moved the increment to the top of both outer addURLToDownload entry chains so every attempted item is counted exactly once regardless of which internal path handles or skips it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Log4j2 docs recommend against modifying Appenders, and recommend building new Configuration objects instead
Moves the single EmptyBorder(5,5,5,5) to the outer content pane instead of applying it separately to every sub-panel, and fixes button preferred sizes so they don't shift position/size when their label text bolds/unbolds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closed
12 tasks
Collaborator
Author
|
@iqqu merged your changes with this pull request, please check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Category
This change is exactly one of the following (please change
[ ]to[x]) to indicate which:Description
Please add details about your change here.
Testing
Required verification:
gradlew test(there are no new failures or errors).Optional but recommended: