Skip to content

feat: add ja.soraraw - #668

Merged
kkantan merged 12 commits into
Aidoku-Community:mainfrom
kinboy56:soraraw.com
Aug 6, 2026
Merged

feat: add ja.soraraw#668
kkantan merged 12 commits into
Aidoku-Community:mainfrom
kinboy56:soraraw.com

Conversation

@kinboy56

@kinboy56 kinboy56 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Search, browsing/filtering, series details, chapter lists, and page lists
  • Three listing types: 新着 (newest), 人気 (hot), ランキング (trending), named after the site's own
    sections
  • Two search paths: a query, which runs over title, alternative title and author the way the site's
    own search does, and the author filter supportsAuthorSearch enables, which matches the author
    alone
  • Dynamic genre filtering, fetched from the site rather than hardcoded
  • Deep linking capabilities for series and chapter URLs

Implementation Notes

  • Search walks the catalogue dump the site publishes, because nothing else on it can be
    queried. /search?q= is statically generated and ignores the query — the browser downloads a
    catalogue and filters it itself. The api host does expose /search, but it answers 500 for every
    query (Unknown column 'Manga.number_views' in 'ORDER BY'), and /mangas ignores every query
    parameter it was tried with. The dump is 13 pages of 2000 entries, about 4.7 MB over the wire
    with the gzip the host serves. The walk stops at 50 matches, since matching is a substring test
    over around 26000 entries. A page that can't be read ends the walk and returns what matched
    before it — the 404 the dump ends with is its terminator, and that is how the site's own search
    stops walking it. The first page is the exception: nothing has been collected there yet, so a
    page 1 that can't be read fails the search rather than returning an empty result that reads as
    "no matches".
  • The author filter narrows the same walk, matching the author column only, and is ANDed with
    the query when both arrive. It comes in as a text filter value rather than as part of the query,
    so it is read out of the filter list. hidesFiltersWhileSearching doesn't collide with it — that
    setting hides filters while a query is present, and an author search arrives with no query.
  • Search cannot be combined with the genre filter, since it walks the whole catalogue rather
    than asking the site for a page, which is what config.hidesFiltersWhileSearching is set for.
  • The reader is picked from genres, not from the mode field. mode describes how the site's
    own reader lays a series out, not what kind of comic it is: across 31 series measured, every
    horizontal one held page-shaped art, but so did a third of the vertical ones — ordinary
    japanese manga that would otherwise open in a continuous scroll. The overseas genres track the
    content instead, appearing on 2 of 7021 horizontal series and 644 of 979 vertical ones across
    8000 catalogue entries, which matches the share of vertical series that really are webtoons.
  • Image proportions are deliberately not used to pick the reader. Korean webtoons on this site
    are cut into page-shaped chunks about as often as into tall strips, so the shape of a page says
    nothing about whether the panels are meant to run together.
  • Page paths are decrypted rather than guessed. Every entry in the page list carries the file
    name on the image server in a b field, encrypted with AES-256-CTR under a key the site's
    scripts derive from the chapter uuid. An earlier revision of this PR rebuilt the names from the
    order and id fields, which are handed out in the clear, and probed the extension with a HEAD
    request per chapter — of 50 chapters sampled, 47 held webp and 3 held jpg. That guess was wrong
    often enough to 404 whole chapters, so decrypt_path now reads the real name out of b. This
    drops one request per chapter and makes the extension exact instead of statistical.
  • Content ratings follow the site's own adult flag. Deriving anything further from genres was
    tried and dropped: genre names are not unique (41 of the 1834 the site lists are used by more
    than one genre), and the ones that read as suggestive are already flagged as adult by the site
    itself, so a name-based guess disagreed with the site more often than it added anything.
  • Publishing status may be absent from a response, which maps to MangaStatus::Unknown rather
    than falling back to ongoing.
  • Chapter language is intentionally omitted to prevent filtering issues on a single-language
    source.

Known limitation: chapters served as one stacked image

A small share of chapters are served as a single image holding every page of the chapter stacked on
top of each other, taller than the reader can draw. This source refuses those chapters with an
error naming the height, rather than handing back a page that renders blank.

Measured on "BLUE GIANT MOMENTUM":

Normal chapters 24 images of 1448x2048
Stacked chapters one image of 1450x49152 — 24 pages of 2048px stacked

At least 5 of that series' 71 chapters are affected, and other series are too — "となりの黒川さん" serves
chapter 49 as one 800x24003 image holding 21 pages. Nothing in the response separates the two
cases: every chapter carries mode: "image", and a stacked chapter lists exactly one page named
001_{id} like any short chapter would, so only the shape of the image itself gives it away.
get_page_list therefore measures a chapter only when it holds four images or fewer, reading the
height off the first 16 KB with a Range request; ordinary chapters take no extra request at all.

Why the pages are not split in the source

Splitting was implemented and works up to the point of display: the page count comes out right and
the cuts land exactly on the page boundaries — scanning the difference between neighbouring pixel
rows around every expected cut put all 23 boundaries of the blue giant image at an offset of 0 (row
difference 88.5 at the cuts against 15.5 inside a page), and all 20 of the kurokawa image likewise.
On device every slice still rendered blank, whether the slices were returned as PageContent::Image
from get_page_list or cut in a PageImageProcessor.

The cause turned out to be in the host rather than in this source, and is filed as
Aidoku/AidokuRunner#3: Canvas.copyImage and drawImage convert the destination rect using the
source image's height where they need the destination context's height, so a canvas shorter
than the image it draws from receives nothing — and the call still reports success. Confirming it
took a throwaway build that passed dst_y = image height - canvas height to cancel the conversion
out: the compensated slices came back with real pixels (2–3 MB each, varying by offset) while the
slices asking for dst_y = 0 stayed at the same 52176 bytes of flat colour. That also rules out
the reading I had assumed first, that a 49152 px image simply fails to decode.

Compensating for the bug in this source would break the moment the host is fixed, so the slicing is
left out and the affected chapters fail loudly instead. If AidokuRunner#3 lands, restoring it is a
small change — the cut geometry is already known to be right, and the code comment on
check_drawable says as much.

Open questions

A few implementation choices were made without being able to fully verify them against the site's
behavior, and are flagged here for reviewers:

  • PATH_SECRET's first 8 digits read as a date (20250805). If the site rotates this value
    periodically, decryption would start failing for everyone at once on some future date. I could not
    confirm whether the value is fixed or rotates, so it is currently hardcoded.
  • MAX_DRAWABLE_HEIGHT = 16384 is a holdover from an earlier (and since disproven) theory that
    the failure was a GPU texture size limit. The real cause turned out to be the AidokuRunner Canvas
    bug described above, so this threshold is not backed by a measurement and may be too conservative
    or too permissive.
  • STRIP_IMAGE_LIMIT = 4 bounds how many images a chapter can hold before it's considered for
    height measurement. A stacked chapter with more than 4 listed images would skip the check and
    still reach the reader as a blank page, undetected.

Test Plan

18 tests, kept in src/test.rs, cover the functionality against the live site:

  • All three listing types and their pagination
  • Search by query, matching a series by both its japanese title and its romanised alternative title
  • Search by the author filter, both against the live catalogue and as a pure-function test pinning
    it to the author column — a walk over the live dump can't tell that apart from a query, since a
    query matches authors too
  • Genre filter parsing and the dynamic filter list
  • Series metadata, and the reader and content rating picked for a webtoon, for an ordinary manga
    the site marks as vertical, and for an adult series
  • Page lists, non-integer chapter numbers, and a malformed chapter key
  • The payload decoding, the path decryption and the jpeg header read, checked without touching the
    network
  • Deep link parsing with edge cases
  • Two stacked chapters from different series, asserting that they fail with the height in the
    message — reaching that point also proves the decrypted path resolved and the Range read
    returned the header

They are unit tests rather than integration tests: the crate is crate-type = ["cdylib"], and
adding rlib so a tests/ binary could use it makes the library itself stop building with
error: #[panic_handler] function required, but not found.

Search and the three listings were also checked on device.

Code quality checks passed: cargo test, cargo fmt, cargo clippy, and aidoku verify.

@kinboy56
kinboy56 marked this pull request as draft August 3, 2026 09:54
Comment thread sources/ja.soraraw/src/lib.rs Outdated
Comment thread sources/ja.soraraw/src/lib.rs Outdated
Comment thread sources/ja.soraraw/src/lib.rs
Comment thread sources/ja.soraraw/src/lib.rs Outdated
Comment thread sources/ja.soraraw/src/lib.rs Outdated
Comment thread sources/ja.soraraw/res/source.json
@kinboy56

kinboy56 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review — all 6 points are addressed.

Two changes go beyond the comments:

  • Page 1 of the catalogue walk is now an error when it can't be read, while later pages break out
    of the loop as before. A search that never reached the dump would otherwise look exactly like a
    search that matched nothing.
  • The doc comment on PageImage was stale in a way that would mislead: it described the page paths
    as being rebuilt from the unencrypted fields, which is what an earlier revision did before
    decrypt_path, and said the d mirror answers for none of the chapters tried. It does answer —
    it just isn't listed for every page, which is the actual reason it goes unread.

Two things the description left out, now added to it:

  • Search stops at 50 matches. The dump holds around 26000 entries and matching is a substring
    test, so a common word would otherwise walk all 13 pages and hand the app thousands of entries.
  • It said 14 tests, which was stale. The suite is 20.

Search and the three listings were also checked on device.

Comment thread sources/ja.soraraw/res/source.json
Comment thread sources/ja.soraraw/src/lib.rs Outdated
Comment thread sources/ja.soraraw/src/lib.rs Outdated
Comment thread sources/ja.soraraw/src/lib.rs Outdated
Comment thread sources/ja.soraraw/src/lib.rs Outdated
@kkantan
kkantan merged commit 439e1a1 into Aidoku-Community:main Aug 6, 2026
2 checks passed
@kinboy56
kinboy56 deleted the soraraw.com branch August 7, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants