Replace per-file replication with periodic cron job to copy entire bucket at once - #3577
Open
tw4l wants to merge 27 commits into
Open
Replace per-file replication with periodic cron job to copy entire bucket at once#3577tw4l wants to merge 27 commits into
tw4l wants to merge 27 commits into
Conversation
tw4l
force-pushed
the
issue-3342-replication
branch
from
August 20, 2026 21:07
d770c54 to
38b7cf7
Compare
tw4l
commented
Aug 20, 2026
tw4l
force-pushed
the
issue-3342-replication
branch
from
August 24, 2026 20:49
aa6828a to
148fb8c
Compare
tw4l
commented
Aug 25, 2026
| spec: | ||
| concurrencyPolicy: Forbid | ||
| successfulJobsHistoryLimit: 0 | ||
| successfulJobsHistoryLimit: 1 |
Member
Author
There was a problem hiding this comment.
Suggested change
| successfulJobsHistoryLimit: 1 | |
| successfulJobsHistoryLimit: 0 |
To reset before merging, but useful for testing
tw4l
commented
Aug 25, 2026
| spec: | ||
| ttlSecondsAfterFinished: 0 | ||
| backoffLimit: 3 | ||
| ttlSecondsAfterFinished: 300 |
Member
Author
There was a problem hiding this comment.
Suggested change
| ttlSecondsAfterFinished: 300 | |
| ttlSecondsAfterFinished: 0 |
To set to 0 before merging, but higher value useful for testing
tw4l
marked this pull request as ready for review
August 25, 2026 20:31
tw4l
commented
Aug 25, 2026
|
|
||
| # Option to disable stuck uploads cron job | ||
| # Generally should only be set true for testing in CI | ||
| disable_stuck_uploads_cron: false |
Member
Author
There was a problem hiding this comment.
We may want to drop the commit that introduced this setting, as it doesn't seem to be making a difference for the nightly tests
Also removes unused arguments from run_delete_replica_job
This option resulted in the background job being marked as failed any time rclone copy ran successfully but there were no new files to copy over.
Also remove replicas from backend tests
It was in the wrong place anyway, and we always want to retry jobs, at least for now.
tw4l
force-pushed
the
issue-3342-replication
branch
from
September 1, 2026 13:03
53c19e7 to
a27e446
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3342
Changes
replicate-files-croncron job that runs on a configurable schedule (current default: every 2 hours) if any replica storage locations are configured on the instance, and launches a newcopy-bucketbackground job for each configured replica locationcopy-bucketbackground job that runsrclone copyon the entire primary storage bucket to a replica storage location. This will sync the replica storage with all files currently in primary without the possibility of deleting anything from the replica destination asrclone syncwould, as replica deletions will still be per-file and subject to the replica deletion delayTo note
A few nightly tests are failing (see most recent run), I believe possibly because the copy bucket job possibly doesn't have sufficient resources to start in our CI runners. We'll want to fix or change the tests before merging, but I could use another pair of eyes on the problem.
Testing
New instance
Existing instance