Added resilient uploads - #106
Open
HeDo88TH wants to merge 15 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR centralizes Dropzone upload retry/backoff and backpressure behavior into a shared useResilientUpload composable, then refactors Upload.vue and DatasetUpload.vue to use it for more consistent handling of transient failures (including Retry-After) and adaptive concurrency (AIMD).
Changes:
- Added
useResilientUpload.jsimplementing retryable-status detection, capped full-jitter exponential backoff withRetry-After, and AIMD concurrency control. - Refactored
Upload.vueandDatasetUpload.vueto use the shared policy and dynamically update DropzoneparallelUploads. - Added
useResilientUpload.spec.jsunit tests covering status classification, backoff behavior,Retry-Afterparsing, and AIMD behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| webapp/js/features/upload/Upload.vue | Integrates shared resilience policy into share-upload flow, including AIMD and retry scheduling. |
| webapp/js/features/dataset/upload/DatasetUpload.vue | Integrates shared resilience policy into dataset upload flow, keeping a bounded legacy retry loop for small files. |
| webapp/js/composables/useResilientUpload.js | New shared composable providing standardized retry/backoff and AIMD concurrency logic. |
| webapp/js/composables/useResilientUpload.spec.js | New unit tests validating the composable’s policy functions and AIMD controller. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
A permanently failed file returned early without advancing the queue, and queuecomplete only reset the uploading state on full success: with autoProcessQueue off, queued files stayed abandoned and the spinner never cleared. Advance the queue in the !canRetry branch, treat queuecomplete as terminal only when nothing is left queued or in flight, and clear the uploading state there. Commit still happens only on full success. Ref: #106
computeRetryDelayMs expects a 0-based attempt, but both upload views incremented the counter before the call, which put the first retry in the 2x base backoff band instead of base. Compute the delay before incrementing in Upload.vue and DatasetUpload.vue. Ref: #106
onFailure() floored concurrency at 2, which could exceed a configured value of 1, contradicting the contract that concurrency never exceeds the configured value. Cap the decreased value at configured; the useResilientUpload spec still passes (16/16). Ref: #106
Tasks that report no progress (null/0/negative percent) should render an animated indeterminate bar until a real percent arrives. Add taskProgressMode() in libs/utils with unit tests and expose it as a method on the useHeavyTask composable for template use.
The Progress column only rendered a bar while active, leaving finished tasks blank and failed tasks showing error text with no bar. Now Processing/Succeeded/Failed all render a bar: ended tasks show a full determinate bar, Processing shows determinate only with a real percent (indeterminate otherwise), and Failed shows the error message inline under the bar. Queued and deleted states keep the muted placeholder. Add a spec covering the rendering rules and PrimeVue ProgressBar DOM output.
COPC files lacking a usable CRS have no WGS84 footprint stored in the DB, so the 3D viewer refused to open them. Exempt point clouds from the geometry gate, detect an unknown CoordinateSystem in loadPointCloud, render the cloud in its local metric coordinates without a basemap, and show a "Local coordinates" badge to explain the display mode.
e.key is undefined on some browsers and devices (e.g. iOS virtual keyboards, synthetic events), which made the shortcut dispatcher throw and broke keyboard shortcuts.
Consumers that outlive the dataset screen (e.g. header bulk download, heavy task tracking) must keep the shared poller alive, so start/stop becomes refcounted acquire/release and the snapshot is discarded on the final release. Read accessors no longer resurrect released stores, and a new tasksUpdated event fires only when the snapshot actually changed, so views can re-render without a full table reload. BuildManager migrates to the refcounted API with a dedicated unregisterDataset.
Unmounting mid-tick rescheduled timers, leaked taskMonitor references and made heavy-task dedup racy; the Tasks tab never refreshed while open and filter changes mishandled pagination. Dataset content now remounts on org/ds change (contentKey) so unmount cleanups actually run.
Mask Borders previously started the job immediately. This adds an explanatory confirmation dialog with a documentation link, consistent with the Align dialog. The file check and job start are unchanged, now run only after confirmation.
Start, cancel, retry, clear and delete left the shared task monitor store stale, so task statuses could lag behind reality. Each operation now waits for a monitor refresh before reloading. Also shows a success toast when tasks are cleared.
Adds an info banner to the Align dialog explaining similarity vs translation modes, the new _aligned output file naming, and linking to the Raster Alignment documentation.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 7 comments.
Suppressed comments (1)
webapp/js/features/dataset/upload/DatasetUpload.vue:357
- The policy retry is visible to Dropzone as
QUEUEDbeforedelayelapses. Dropzone emitscompleteright aftererror, and line 431 schedulesprocessQueue()in 100 ms, so this retry—including a server-directedRetry-After—starts almost immediately. Keep it non-queued until this timer expires, then transition it toQUEUEDand process the queue.
file.status = Dropzone.QUEUED;
this.scheduleProcessQueue(delay);
|
|
||
| // Update progress | ||
| this.totalBytesSent = this.totalBytesSent - file.trackedBytesSent; | ||
| file.status = Dropzone.QUEUED; |
| function computeSignature(ent) { | ||
| return Array.from(ent.tasks.values()) | ||
| .sort((a, b) => String(a.taskId).localeCompare(String(b.taskId))) | ||
| .map(t => `${t.taskId}:${t.state}:${t.progressPercent ?? ''}:${t.phaseMessage ?? ''}`) |
| this._backgroundReload(); | ||
| } | ||
| }; | ||
| taskMonitor.on('tasksUpdated', this._onTasksUpdated); |
Comment on lines
+17
to
+18
| <label class="section-label">Source file</label> | ||
| <InputText :modelValue="entry.entry.path" readonly fluid class="w-100" /> |
| /** | ||
| * Additive-increase / multiplicative-decrease concurrency controller. | ||
| * | ||
| * on503(): halves concurrency (floor 2). |
Comment on lines
+109
to
+114
| computed: { | ||
| // Forces a remount when switching dataset: /r/:org/:ds is a single route | ||
| // record, so without a key Vue would reuse the instance and skip its hooks. | ||
| contentKey() { | ||
| const p = this.$route.params; | ||
| return p.org && p.ds ? `${p.org}/${p.ds}` : this.$route.path; |
Comment on lines
+343
to
+344
| file.status = Dropzone.QUEUED; | ||
| this.scheduleProcessQueue(delay); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a shared, robust retry and backpressure policy for Dropzone-based file uploads, encapsulated in the new
useResilientUploadcomposable. BothDatasetUpload.vueandUpload.vueare refactored to use this composable, eliminating duplicated ad-hoc retry logic and improving reliability, especially under transient server errors and rate-limiting. The changes also add adaptive concurrency control (AIMD) and respect server-providedRetry-Afterheaders for coordinated backoff. Comprehensive unit tests are included for the new composable.New shared upload resilience policy:
useResilientUpload.js, which provides:Retry-Aftersupport,useResilientUpload.spec.jswith thorough unit tests for all resilience policy behaviors.Refactoring and integration in upload components:
DatasetUpload.vue:useResilientUploadfor all but legacy small-file retry logic.parallelUploadsdynamically based on AIMD controller feedback.Retry-Afterfor retries, with clear separation between legacy small-file and policy-based retries.Upload.vue:useResilientUpload, removing custom retry logic and hardcoded retry limits.parallelUploadsaccordingly. [1] [2] [3] [4] [5] [6]These changes centralize and standardize upload retry and backoff logic, making future maintenance easier and improving upload robustness under adverse network/server conditions.