feat(par2repair): background PAR2 repair of missing usenet articles - #829
Open
javi11 wants to merge 62 commits into
Open
feat(par2repair): background PAR2 repair of missing usenet articles#829javi11 wants to merge 62 commits into
javi11 wants to merge 62 commits into
Conversation
…spec compatibility
…persistence, patch-store cap
The repair reason was stored and served but rendered as a truncated line under the filename, so in practice an unrepairable job showed a red badge and nothing else. Give the reason its own full-width row and translate the solver's wording into what the user can act on: not enough recovery data posted (replace the release) vs. damage above the configured limit (raise it in settings). Also clear last_error when a job finally succeeds — it is no longer true of the job, and the UI now reports that column as a failure.
Deriving a file's decoded part size required probing one of its own live articles, because an NZB records only yEnc-encoded byte counts. A volume with no live article therefore aborted the whole repair -- excluding exactly the case PAR2 repair exists for, a fully missing volume. Borrow the part size from a sibling volume instead: a usenet post splits every file of a release at one uniform size. The borrowed layout is validated against the file's exact PAR2 length, and every rebuilt slice is already verified against its PAR2 MD5 before being stored, so a wrong layout fails the repair rather than corrupting it.
…ining time A repair streams the entire release, so it runs for minutes to hours -- the duration is the real cost the user trades for the recovered bytes, and a bare percentage does not say whether to wait for it. Record started_at on every claim (a retry re-runs the whole sweep, so the clock belongs to the current attempt) and finished_at on both terminal outcomes. The UI shows elapsed time plus an ETA extrapolated from the sweep rate while running, and the total for finished jobs -- including failed ones, since a repair must download the release before it can tell that the damage exceeds its recovery data.
Repairs are started by imports, playback and health checks, none of which the Health page hears about, so polling is the list's only way to learn a job exists. That poll paused whenever the tab lost focus (refetchIntervalInBackground defaults to false) and the app disables focus refetching globally, so looking away froze the list until a manual reload. Refetch on focus and drop the idle interval to 5s. staleTime must be 0 for the focus refetch to fire at all -- the global 5 minute staleTime would otherwise skip it.
…story A 900MB release took ~5 minutes to report that it could not be repaired, and a 372-block repair refused to run at all over the memory budget. Reach a verdict before downloading the release: - Add an optional ArticleStater capability to ArticleFetcher, implemented on PoolFetcher over the pool's StatMany. Both resolvers now sweep every article's liveness with bodyless STATs before planning, so the missing set is exact up front instead of being discovered mid-sweep (each discovery previously cost a full replan-and-retry cycle). - Drop recovery slices whose backing PAR2 articles are dead before BuildPlan. ParseIndex seeks past payloads without fetching them, so such a slice used to parse fine and only fail deep into the job. - Reject damage far above max_repair_ratio right after the STAT sweep, estimated from the NZB layout alone. The estimate carries a 5% margin so borderline releases still fall through to BuildPlan's exact check. Keep the network busy while sweeping: - Sweep and recovery-slice fetches now run through a bounded ordered prefetch window (8 in flight) instead of one article at a time. ArticleFetcher is documented as concurrency-safe accordingly. - Bound the resolve/job article caches (FIFO, 64 entries). They previously grew with the recovery set, pinning hundreds of MB for a large release. Run large repairs on disk rather than refusing them: - MaxMemoryMB becomes a spill threshold, not a hard cap. Over-budget plans set Plan.SpillToDisk and back accumulators, recovery payloads and recovered slices with memory-mapped scratch files (unix + windows shims) under the patch store, cleaned up per job and swept at startup. Stop accumulating job rows: - Terminal outcomes are translated to where users already look — the file's health record for imported files, the import queue entry for NZB-mode repairs — and the job row is deleted. Only pending/running states remain; a startup sweep clears terminal rows left by older versions. The health table renders repair verdicts in their translated, actionable form.
Repaired article payloads and the solver's memory-mapped scratch files were pinned to <metadata_root>/patches. A large repair can need gigabytes of scratch, which is not necessarily where the metadata lives. Add par2_repair.patch_dir, exposed in Settings -> PAR2 Repair. Empty keeps the previous location, so existing installs are unaffected. Applied at startup; existing patches are not moved (they regenerate on demand).
The feature is new and each repair downloads a full release in the background, so an install should not start doing that unasked. Default par2_repair.enabled to false and add a Beta banner to the config section explaining the cost, so enabling it is a deliberate choice.
BuildPlan charged the repair-ratio cap for k * sliceSize — the bytes the solver rebuilds — instead of the bytes actually missing. Releases post PAR2 slices far larger than their articles, so every dead article marked a whole slice missing and inflated the measured damage by the slice/article factor. A real season pack showed the effect: 27.3 MB slices over 1.05 MB articles (26x), ~9 missing articles per episode. Health reported 0.1% of the file missing; the planner computed a 9.75% damage ratio and refused to repair against a 5% cap. The reported percentage was wrong too — the UI said "9.75% of this file is missing" when 0.1% was. Count missing article bytes for the ratio and keep k * sliceSize solely for the solver memory/spill decision, which genuinely scales with slices.
Three throughput and resilience improvements to the repair pipeline: - FoldPresent splits slices >=128KiB across GOMAXPROCS workers by byte range, keeping each span cache-hot across all accumulator rows (~4x fold throughput on an M4: 1752 vs 440 MB/s). - Plans carry up to 8 margin recovery rows beyond the damage. Margin rows absorb mid-sweep surprises in the same pass: articles that died after planning are patched in the same job, corrupt present slices no longer force a re-sweep, and Solve retries a singular matrix by selecting a linearly independent row subset instead of re-sweeping. Margin shrinks rather than push an in-memory job into disk spill. - Recovery payload buffers are donated to the solver as accumulators (SeedRecoveryOwning), eliminating the per-row copy and the separate payload set; the rare fallback replans refetch payloads instead. All fallback paths (margin exhausted, no invertible subset) keep the previous replan/typed-error behavior, so the service contract is unchanged.
Replace akalin/gopar with javi11/gopar-turbo v0.1.0 and run the solver's region arithmetic on its gf16 backend (ParPar SIMD kernels under cgo, pure-Go fallback otherwise): - Accumulators live in the backend's prepared layout for the whole fold and are untransformed once, in Solve. They are still carved out of the caller's allocator (page-aligned), so disk-spill arenas keep bounding memory. - The parallel fold keeps its stride-aligned byte-range split but gives each worker its own gf16 context; folds go through MulAddMulti. - SeedRecoveryOwning now consumes the donated buffer into the prepared accumulator (zero-copy seeding is impossible in prepared layout); the spill arena grows to hold payloads, accumulators and outputs. - Solver gains Close() for deterministic context release per attempt. BenchmarkFoldPresent on an M4: 1580 -> 4573 MB/s (2.9x). Also bumps the module to Go 1.27.
…epair # Conflicts: # cmd/altmount/cmd/serve.go
Contributor
Review: security & correctness pass 👋Hi! We're reviewing this closely because our PR (#839) also modifies First, the good stuff — several things here are done really well: routes correctly placed inside the JWT-protected A few things worth addressing:
Minor: SQLite migrations 036–038 lack the Coordination note: #839 optimizes probing inside |
Corrupted files still go to Radarr/Sonarr first, unchanged. When the ARRs come up empty -- no instance tracks the file, none configured, or ARR repair disabled -- the health worker now queues a PAR2 repair instead of finalizing the file as corrupted. Gated by par2_repair.arr_first (default on); degraded files, playback holes and repair-on-import keep triggering PAR2 directly. Also in this branch: a repair connection limiter (par2_repair.max_connections) so repairs no longer borrow the import budget, min/max release size caps, release-grouped repair jobs (migration 039) so one job covers every damaged file of an NZB, and the matching Health page and config UI.
Adds an NZB-mode verify sweep so a release whose articles download fine but no longer decode is repaired instead of failed, and fixes two ways a repairable release could still be dropped. Verify sweep (Caps.VerifySweep): builds a plan with no known-missing slices — margin rows only — so the job's CRC sweep locates the corrupt slices itself and absorbs them onto the margin. Import defers into it when rardecode corruption sentinels break archive analysis, which the fast-fail sweep cannot see (it only observes MISSING articles). fix(importer): also defer when analysis hits a MISSING article. The fast-fail release probe samples only import.segment_sample_percentage of segments, so it can pass clean on a release that does have a dead article; the miss then surfaces mid-analysis and was treated as terminal. Same damage the escalation path already repairs, so defer it too. fix(par2repair): report every article a sweep proved dead, not just the last one. SweepDeadArticleError carried a single message ID, so the ones already absorbed onto margin rows were discarded with the error. Each retry then re-parsed the PAR2 set and re-read the whole release to learn one new article, so a release with more dead articles than margin rows exhausted maxJobAttempts instead of converging. Absorbed/DeadMessageIDs carry them all out and the service persists them for the next plan. Also bumps gopar-turbo to v0.2.0.
Resolves a textual conflict and a semantic one in the fast-fail sweep. fast_fail.go (textual): both sides changed the StatMany error branch and the changes are complementary, so both are kept. The local-patch check runs first — a locally repaired article is available regardless of what the provider Stat said — then main's caller-cancellation check distinguishes the caller giving up from the probe's own short deadline expiring, which is a reachability signal. The patched branch now also increments main's new `reachable` counter. HEAD had no such counter, so `continue` was enough there; after the merge the unproven-sample check compares reachable against len(ids), and skipping the increment reported a fully patched release as missing (caught by TestReleaseProbeTreatsPatchedSegmentAsAvailable). fast_fail_test.go (semantic): main appended three tests calling FastFailReleaseProbe/FastFailCheckFiles with the pre-branch signatures, while this branch had widened both with a trailing PatchIndex. The file merged cleanly but did not compile — this is the CI "test" failure. The new tests exercise timeout and cancellation, not patch awareness, so they pass a nil PatchIndex.
…weep
FastFailCheckFiles read per-segment outcomes out of a map populated only
from the results StatMany actually yielded. StatMany abandons pending
sends once its chunk context is done, so a deadline expiry produces NO
result for the segments still in flight — and an absent map entry reads
back as a nil error, i.e. reachable. Segments that never proved reachable
were therefore imported as healthy.
The surrounding comment already documented the intended contract ("any
error left here — including this chunk's own deadline — is a reachability
signal for the owning file"); only the implementation was missing. Seed
every sampled segment with errStatUnconfirmed so only a result that
actually arrives can clear it, and clear it explicitly for locally
patched segments.
This is what made TestFastFailCheckFilesTimeoutMarksFileBroken flaky —
it failed ~2 in 12 runs on main, depending on whether StatMany won the
race to emit the error before abandoning the send. 20/20 clean now.
… feat/par2-streaming-repair
Contributor
|
@javi11 Very nice. I'd like to see this merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds background PAR2 repair: when a release has missing (taken-down) articles, AltMount now reconstructs the missing bytes with Reed–Solomon from the release's PAR2 recovery files, persists the repaired article payloads locally, and serves them byte-exact on every later read — permanently, surviving restarts. Playback behavior on first hit is unchanged (instant zero-fill for plain video); the bytes now come back afterwards.
How
internal/importer/parser/par2: parsesMain/IFSC/RecvSlicpackets (previously onlyFileDesc); recovery payloads are located, not loaded, via seek-aware lazy readers so planning doesn't download recovery data.internal/par2repair(new):gf2p16: each present input slice is folded into k accumulators and discarded. Memory ≈ damage size, never release size. Singular-matrix retries swap in spare recovery slices (known PAR2 Vandermonde flaw).max_repair_ratio, memory budget, available redundancy).<metadata_root>/patches, atomic tmp+rename writes: readers see a miss or complete verified bytes, never garbage.par2_repair_jobs, migration 035, sqlite+postgres) with dedup, exponential backoff, crash recovery on boot.POST /api/par2repair.Cost model (for reviewers)
Repairing anything requires streaming the whole recovery set once (~one release download) — inherent to PAR2's math, same as any PAR2 tool. It runs once per file, on the import connection lane. Disk cost is damage-sized only (typically a few MB per repaired file).
Test plan
par2gennow generates full spec-valid PAR2 sets: Main/IFSC/RecvSlic with real Vandermonde recovery data, independent of the solver's fold path)-race)go test ./... -racegreenmakelint: fails on 3 pre-existing issues from feat(stremio): add local library stream provider for instant 0s playback #828 (stremio files, untouched by this PR) — tracked separately; all packages touched here lint at 0 issues