Skip to content

Replace per-file replication with periodic cron job to copy entire bucket at once - #3577

Open
tw4l wants to merge 27 commits into
mainfrom
issue-3342-replication
Open

Replace per-file replication with periodic cron job to copy entire bucket at once#3577
tw4l wants to merge 27 commits into
mainfrom
issue-3342-replication

Conversation

@tw4l

@tw4l tw4l commented Aug 14, 2026

Copy link
Copy Markdown
Member

Fixes #3342

Changes

  • Remove per-file replication jobs (keeps model and some code paths required to get out and serialize the old jobs from the database, but they can no longer be run or retried)
  • Add a replicate-files-cron cron job that runs on a configurable schedule (current default: every 2 hours) if any replica storage locations are configured on the instance, and launches a new copy-bucket background job for each configured replica location
  • Add a copy-bucket background job that runs rclone copy on the entire primary storage bucket to a replica storage location. This will sync the replica storage with all files currently in primary without the possibility of deleting anything from the replica destination as rclone sync would, as replica deletions will still be per-file and subject to the replica deletion delay
  • Stop tracking replicas on each file in the database
  • Add a migration to remove per-file replica information from the database
  • Remove "backed up" icons from the frontend

To note

A few nightly tests are failing (see most recent run), I believe possibly because the copy bucket job possibly doesn't have sufficient resources to start in our CI runners. We'll want to fix or change the tests before merging, but I could use another pair of eyes on the problem.

Testing

New instance

  1. Set up a local Browsertrix instance with one or more default replica storage locations configured
  2. Run some crawls and browser profiles
  3. Verify that the cron job runs on the configured schedule and kicks off one copy bucket job per replica location
  4. Verify those jobs are successful
  5. Verify all files are replicated to each default replica storage location

Existing instance

  1. Bootstrap an existing Browsertrix instance that already has at least one replica location configured and has previously had per-file replication jobs run
  2. Verify that the new cron replication and copy bucket jobs run and are successful
  3. Verify that the switch in replication jobs have not modified any filepaths unexpected in replica locations

@tw4l
tw4l force-pushed the issue-3342-replication branch from d770c54 to 38b7cf7 Compare August 20, 2026 21:07
Comment thread backend/btrixcloud/main_bg.py Outdated
spec:
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 0
successfulJobsHistoryLimit: 1

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
successfulJobsHistoryLimit: 1
successfulJobsHistoryLimit: 0

To reset before merging, but useful for testing

spec:
ttlSecondsAfterFinished: 0
backoffLimit: 3
ttlSecondsAfterFinished: 300

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ttlSecondsAfterFinished: 300
ttlSecondsAfterFinished: 0

To set to 0 before merging, but higher value useful for testing

@tw4l
tw4l marked this pull request as ready for review August 25, 2026 20:31
@tw4l
tw4l requested a review from ikreymer August 25, 2026 20:31
Comment thread chart/values.yaml

# Option to disable stuck uploads cron job
# Generally should only be set true for testing in CI
disable_stuck_uploads_cron: false

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to drop the commit that introduced this setting, as it doesn't seem to be making a difference for the nightly tests

@tw4l tw4l changed the title Replace per-file replication with cron job to periodically copy entire bucket Replace per-file replication with periodic cron job to copy entire bucket at once Aug 25, 2026
@tw4l
tw4l force-pushed the issue-3342-replication branch from 53c19e7 to a27e446 Compare September 1, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task]: Switch from per-file replication jobs to periodically syncing replica locations with rclone copy

1 participant