internal/dave/repo_s3: handle upload failures

[joshuasing]

The default logic of Dave is:

1. Clone data directories to Dave home
2. Create archives of cloned data directories
3. Add snapshot containing archives to repository (uploading each archive to S3)
4. Remove local copies of archives to reduce storage requirement

The issue with this logic is that if a part of step 3 fails, Dave may delete all local copies of the archives because it is "done" with the archives. We have encountered cases where uploading very large archives to S3 is unstable, resulting in us needing to manually upload the archives with the aws s3 CLI.

A --keep-archives flag was added in 37db163 as a temporary way to work around this issue and manually upload the archives.

We should improve how we handle these upload failures, making it possible to retry uploading remaining archives if Dave fails. Dave should also retry multiple times to continue uploading files to S3 if it encounters a temporary error.

[ClaytonNorthey92]

talking with @joshuasing on slack...

• add flags --max-retries and --backoff-seconds
• upon upload
  • if the upload fails due to a temporary error (to be defined with investigation), wait --backoff-seconds seconds then retry
  • if the upload fails again, wait current retry**--backoff-seconds seconds (example: 5**2 = 25 seconds for the 2nd retry)
• upon upload failure after max retries
  • check --keep-archives flag, delete if explicitly set to false
• upon success, delete archive unless --keep-archives is not set to true (can be null/unset)
  exit with code based on upload success or failure

[ClaytonNorthey92]

@AL-CT @joshuasing as of now, I consider any error "temporary" in the PR. it was a bit too difficult to distinguish errors, we can change that if y'all would like.