From 9ce8c7c1969802237d3b3b82c7025a91bb71a33d Mon Sep 17 00:00:00 2001
From: Christian Foellmann <christian@foellmann.de>
Date: Sun, 30 Aug 2026 12:50:38 +0200
Subject: [PATCH] Add support for multiple S3 targets per backup job

Backup jobs could only ever be uploaded to a single S3 endpoint. Mirroring a
backup to a second provider meant either a second backup job dumping the same
database twice, or a hand written post script that reimplements the upload,
the checksum handling and the retention.

S3_TARGETS takes a comma separated list of target names and uploads the very
same dump and checksum file to each of them, so all endpoints hold identical
content. Every S3 option exists per target as S3_<TARGET>_<OPTION> and per job
as DB01_S3_<TARGET>_<OPTION>, falling back to the existing single target
variables for everything a target does not override - so shared settings stay
written once. _FILE secrets work for the per target variables as well.

Leaving S3_TARGETS unset keeps the previous behaviour untouched: the target
list then holds a single empty entry and the existing S3_ variables are used.

Every target is attempted even after an earlier one failed, so a broken
endpoint no longer keeps the healthy ones from receiving their backup. A
failure names the target in the log and is aggregated into move_exit_code,
which already drives the failure notification and reaches post scripts as $11.

The S3 retention pass loops over the same targets. It now builds its own
aws-cli environment per target instead of relying on the credentials, endpoint
URL and TLS flags left behind by the preceding upload.

Verified against two MinIO endpoints with the container image: identical
sha256 on both targets, a stopped endpoint reported as an error while the
healthy target still received the backup, retention deleting on both targets,
and an unchanged single target configuration.
---
 CHANGELOG.md                             |  15 ++
 README.md                                |  53 +++++++
 examples/multiple-s3-targets/compose.yml |  76 ++++++++++
 rootfs/container/functions/10-dbbackup   | 175 +++++++++++++++++++----
 4 files changed, 288 insertions(+), 31 deletions(-)
 create mode 100644 examples/multiple-s3-targets/compose.yml

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 12bbd5f..955522f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,18 @@
+## 4.10.0 2026-08-30 <foellmann at wus-technik dot com>
+
+   ### Added
+      - Multiple S3 targets per backup job via `S3_TARGETS`, uploading each backup to every
+        configured endpoint with identical content
+      - Per target configuration as `S3_<TARGET>_<OPTION>` / `DB01_S3_<TARGET>_<OPTION>`, falling
+        back to the existing single target `S3_` variables for everything a target does not override
+      - `_FILE` secret support for the per target variables
+
+   ### Changed
+      - S3 cleanup now sets up its own aws-cli environment per target instead of relying on the
+        values left behind by the preceding upload
+      - A failed S3 upload names the target it failed on and no longer aborts the remaining targets
+
+
 ## 4.9.2 2026-08-24 <code at nfrastack dot com>
 
    ### Changed
diff --git a/README.md b/README.md
index 935c690..03de506 100644
--- a/README.md
+++ b/README.md
@@ -333,6 +333,7 @@ If `DEFAULT_BACKUP_LOCATION` = `S3` then the following options are used:
 
 | Parameter                     | Description                                                                               | Default | `_FILE` |
 | ----------------------------- | ----------------------------------------------------------------------------------------- | ------- | ------- |
+| `DEFAULT_S3_TARGETS`          | Comma separated list of S3 targets to upload each backup to, e.g. `hetzner,impossible`    |         |         |
 | `DEFAULT_S3_BUCKET`           | S3 Bucket name e.g. `mybucket`                                                            |         | x       |
 | `DEFAULT_S3_KEY_ID`           | S3 Key ID (Optional)                                                                      |         | x       |
 | `DEFAULT_S3_KEY_SECRET`       | S3 Key Secret (Optional)                                                                  |         | x       |
@@ -347,6 +348,51 @@ If `DEFAULT_BACKUP_LOCATION` = `S3` then the following options are used:
 
 - When `DEFAULT_S3_KEY_ID` and/or `DEFAULT_S3_KEY_SECRET` is not set, will try to use IAM role assigned (if any) for uploading the backup files to S3 bucket.
 
+**Multiple S3 targets**
+
+Set `DEFAULT_S3_TARGETS` (or `DB01_S3_TARGETS` for a single job) to a comma separated list of
+names to upload every backup to more than one S3 endpoint. All targets receive the very same
+file, so the endpoints hold identical content:
+
+```yaml
+DEFAULT_S3_TARGETS: hetzner,impossible
+DEFAULT_S3_PATH: db                                      ## shared by both targets
+
+S3_HETZNER_HOST: fsn1.your-objectstorage.com
+S3_HETZNER_BUCKET: backups
+S3_HETZNER_KEY_ID: xxxxxxxx
+S3_HETZNER_KEY_SECRET: xxxxxxxx
+S3_HETZNER_REGION: eu-central
+
+S3_IMPOSSIBLE_HOST: eu-central-2.storage.impossibleapi.net
+S3_IMPOSSIBLE_BUCKET: backups-offsite
+S3_IMPOSSIBLE_KEY_ID: yyyyyyyy
+S3_IMPOSSIBLE_KEY_SECRET: yyyyyyyy
+S3_IMPOSSIBLE_REGION: eu-central-2
+```
+
+- Every S3 option above also exists per target as `S3_<TARGET>_<OPTION>` and per job as
+  `DB01_S3_<TARGET>_<OPTION>`. The value used for a target is the first one that is set of:
+
+  `DB01_S3_<TARGET>_<OPTION>` → `S3_<TARGET>_<OPTION>` → `DB01_S3_<OPTION>` → `S3_<OPTION>` → `DEFAULT_S3_<OPTION>`
+
+  Options that all targets share are therefore written once as `DEFAULT_S3_<OPTION>`, and only
+  what actually differs is repeated per target.
+- Target names are case insensitive, and any character that is not a letter or a digit becomes an
+  underscore - a target named `impossible-cloud` is configured through `S3_IMPOSSIBLE_CLOUD_*`.
+- `_FILE` secrets work for the per target variables too, e.g. `S3_HETZNER_KEY_SECRET_FILE`.
+- Every target is attempted, including after an earlier one failed, so a broken endpoint does not
+  stop the healthy ones from receiving their backup. A failed target is logged as an error naming
+  that target, raises the usual `Moving of backup ... reported errors` notification, and is handed
+  to post scripts as `$11` (`MOVE_EXIT_CODE`).
+- An unreachable endpoint is retried by `aws-cli` before it gives up, which can hold up the job for
+  several minutes. Shorten this per target with e.g.
+  `S3_HETZNER_EXTRA_OPTS: --cli-connect-timeout 10 --cli-read-timeout 30`.
+- `DEFAULT_CLEANUP_TIME` applies to all targets alike. Cleanup is skipped for a job that ran into
+  errors, so a failing target also postpones the cleanup on the healthy ones.
+- Leaving `S3_TARGETS` unset keeps the previous single target behaviour of the `DEFAULT_S3_*` and
+  `DB01_S3_*` variables unchanged.
+
 ###### Azure
 
 If `DEFAULT_BACKUP_LOCATION` = `blobxfer` then the following options are used:.
@@ -619,6 +665,7 @@ If `DB01_BACKUP_LOCATION` = `S3` then the following options are used:
 
 | Parameter                  | Description                                                                               | Default | `_FILE` |
 | -------------------------- | ----------------------------------------------------------------------------------------- | ------- | ------- |
+| `DB01_S3_TARGETS`          | Comma separated list of S3 targets to upload each backup to, e.g. `hetzner,impossible`    |         |         |
 | `DB01_S3_BUCKET`           | S3 Bucket name e.g. `mybucket`                                                            |         | x       |
 | `DB01_S3_KEY_ID`           | S3 Key ID (Optional)                                                                      |         | x       |
 | `DB01_S3_KEY_SECRET`       | S3 Key Secret (Optional)                                                                  |         | x       |
@@ -633,6 +680,12 @@ If `DB01_BACKUP_LOCATION` = `S3` then the following options are used:
 
 > When `DB01_S3_KEY_ID` and/or `DB01_S3_KEY_SECRET` is not set, will try to use IAM role assigned (if any) for uploading the backup files to S3 bucket.
 
+**Multiple S3 targets**
+
+Set `DB01_S3_TARGETS` to a comma separated list of names to upload this job's backups to more than
+one S3 endpoint, each target configured through `DB01_S3_<TARGET>_<OPTION>`. See
+[Multiple S3 targets](#s3) under the job defaults for the lookup order and the behaviour on failure.
+
 ###### Azure
 
 If `DB01_BACKUP_LOCATION` = `blobxfer` then the following options are used:.
diff --git a/examples/multiple-s3-targets/compose.yml b/examples/multiple-s3-targets/compose.yml
new file mode 100644
index 0000000..8524394
--- /dev/null
+++ b/examples/multiple-s3-targets/compose.yml
@@ -0,0 +1,76 @@
+#
+# Example for uploading every backup to two S3 endpoints at once
+#
+# Both endpoints receive the identical dump and checksum file. Options that both targets
+# share are written once as DEFAULT_S3_*, only what differs is repeated per target.
+#
+
+services:
+  example-multi-s3-db:
+    hostname: example-db-host
+    image: docker.io/library/mariadb:11
+    container_name: example-multi-s3-db
+    restart: unless-stopped
+    networks:
+      example-multi-s3-net:
+    environment:
+      MARIADB_ROOT_PASSWORD: rootpass
+      MARIADB_DATABASE: example
+
+  # First endpoint - stands in for any S3 compatible service
+  example-multi-s3-minio:
+    hostname: minio
+    image: quay.io/minio/minio:latest
+    container_name: example-multi-s3-minio
+    restart: unless-stopped
+    command: server /data
+    networks:
+      example-multi-s3-net:
+    environment:
+      MINIO_ROOT_USER: miniokey
+      MINIO_ROOT_PASSWORD: miniosecret
+
+  example-multi-s3-db-backup:
+    container_name: example-multi-s3-db-backup
+    image: nfrastack/db-backup
+    restart: always
+    networks:
+      example-multi-s3-net:
+    environment:
+      - TIMEZONE=Europe/Berlin
+      - CONTAINER_ENABLE_MONITORING=FALSE
+      - CONTAINER_NAME=example-multi-s3-db-backup
+      # - DEBUG_MODE=TRUE
+
+      - DB01_TYPE=mysql
+      - DB01_HOST=example-db-host
+      - DB01_NAME=example
+      - DB01_USER=root
+      - DB01_PASS=rootpass
+      - DB01_BACKUP_INTERVAL=1           # backup every minute
+      - DB01_CLEANUP_TIME=60             # applies to every target alike
+
+      - DEFAULT_BACKUP_LOCATION=S3
+      - DEFAULT_S3_PATH=example          # shared by both targets
+
+      # Upload to both targets, in this order
+      - DEFAULT_S3_TARGETS=local,offsite
+
+      - S3_LOCAL_HOST=minio:9000
+      - S3_LOCAL_PROTOCOL=http
+      - S3_LOCAL_BUCKET=backups
+      - S3_LOCAL_KEY_ID=miniokey
+      - S3_LOCAL_KEY_SECRET=miniosecret
+      - S3_LOCAL_REGION=us-east-1
+
+      - S3_OFFSITE_HOST=s3.example.com
+      - S3_OFFSITE_BUCKET=backups-offsite
+      - S3_OFFSITE_KEY_ID=xxxxxxxx
+      - S3_OFFSITE_KEY_SECRET=xxxxxxxx
+      - S3_OFFSITE_REGION=eu-central-1
+      # Do not let an unreachable offsite endpoint stall the job for minutes
+      - S3_OFFSITE_EXTRA_OPTS=--cli-connect-timeout 10 --cli-read-timeout 30
+
+networks:
+  example-multi-s3-net:
+    name: example-multi-s3-net
diff --git a/rootfs/container/functions/10-dbbackup b/rootfs/container/functions/10-dbbackup
index 91ecaa6..175043c 100644
--- a/rootfs/container/functions/10-dbbackup
+++ b/rootfs/container/functions/10-dbbackup
@@ -112,6 +112,23 @@ dbbackup_bootstrap_variables() {
                             S3_PROTOCOL \
                             S3_EXTRA_OPTS
                                                                                   ## Legacy after DEFAULT
+
+        ## Multiple S3 targets - resolve the target list straight from the environment so
+        ## that the per target variables can take part in the '_FILE' secret handling too.
+        s3_target_list_var="DB${backup_instance_number}_S3_TARGETS"
+        s3_target_list="${!s3_target_list_var:-${S3_TARGETS:-${DEFAULT_S3_TARGETS:-}}}"
+        if [ -n "${s3_target_list}" ] ; then
+            s3_target_file_vars=""
+            for s3_target_entry in ${s3_target_list//,/ } ; do
+                s3_target_id="${s3_target_entry^^}"
+                s3_target_id="${s3_target_id//[^A-Z0-9]/_}"
+                for s3_target_key in BUCKET CERT_CA_FILE EXTRA_OPTS HOST KEY_ID KEY_SECRET PATH PROTOCOL REGION ; do
+                    s3_target_file_vars="${s3_target_file_vars} DB${backup_instance_number}_S3_${s3_target_id}_${s3_target_key} S3_${s3_target_id}_${s3_target_key}"
+                done
+            done
+            transform_var file ${s3_target_file_vars}
+        fi
+
         set -o posix ; set | grep -E "^DB${backup_instance_number}_|^DEFAULT_|^DB_|^ARCHIVE|^BACKUP_|^BLOBXFER_|^CHECKSUM|^COMPRESSION|^CREATE_|^ENABLE_|^EXTRA_|^GZ_|^INFLUX_|^MYSQL_|^MONGO_|^PARALLEL|^PRE_|^POST_|^S3|^SKIP|^SPLIT" > "${backup_instance_vars}"
 
         ##
@@ -222,6 +239,7 @@ dbbackup_bootstrap_variables() {
         transform_backup_instance_variable "${backup_instance_number}" S3_PATH backup_job_s3_path
         transform_backup_instance_variable "${backup_instance_number}" S3_PROTOCOL backup_job_s3_protocol
         transform_backup_instance_variable "${backup_instance_number}" S3_REGION backup_job_s3_region
+        transform_backup_instance_variable "${backup_instance_number}" S3_TARGETS backup_job_s3_targets
         transform_backup_instance_variable "${backup_instance_number}" SCRIPT_LOCATION_POST backup_job_script_location_post
         transform_backup_instance_variable "${backup_instance_number}" SCRIPT_LOCATION_PRE backup_job_script_location_pre
         transform_backup_instance_variable "${backup_instance_number}" SIZE_VALUE backup_job_size_value
@@ -230,6 +248,38 @@ dbbackup_bootstrap_variables() {
         transform_backup_instance_variable "${backup_instance_number}" TYPE backup_job_db_type
         transform_backup_instance_variable "${backup_instance_number}" USER backup_job_db_user
 
+
+        ## Multiple S3 targets - resolve every target listed in S3_TARGETS into its own
+        ## set of backup_job_s3_<target>_<key> variables. Precedence per key:
+        ## DB<NN>_S3_<TARGET>_<KEY> > S3_<TARGET>_<KEY> > the single target variables
+        ## handled by transform_backup_instance_variable.
+        transform_s3_target_variable() {
+            local s3_t_instance="${1}"
+            local s3_t_target="${2}"
+            local s3_t_key="${3}"
+            local s3_t_out="${4}"
+            unset "${s3_t_out}"
+            if grep -q "^DB${s3_t_instance}_S3_${s3_t_target}_${s3_t_key}=" "${backup_instance_vars}" && [ "$(grep "^DB${s3_t_instance}_S3_${s3_t_target}_${s3_t_key}=" "${backup_instance_vars}" | cut -d = -f2)" != "unset" ]; then
+                export "${s3_t_out}"="$(grep "^DB${s3_t_instance}_S3_${s3_t_target}_${s3_t_key}=" "${backup_instance_vars}" | cut -d = -f2-)"
+            elif grep -q "^S3_${s3_t_target}_${s3_t_key}=" "${backup_instance_vars}" && [ "$(grep "^S3_${s3_t_target}_${s3_t_key}=" "${backup_instance_vars}" | cut -d = -f2)" != "unset" ]; then
+                export "${s3_t_out}"="$(grep "^S3_${s3_t_target}_${s3_t_key}=" "${backup_instance_vars}" | cut -d = -f2-)"
+            else
+                transform_backup_instance_variable "${s3_t_instance}" "S3_${s3_t_key}" "${s3_t_out}"
+            fi
+        }
+
+        if [ -n "${backup_job_s3_targets}" ] ; then
+            for s3_target_entry in ${backup_job_s3_targets//,/ } ; do
+                s3_target_id="${s3_target_entry^^}"
+                s3_target_id="${s3_target_id//[^A-Z0-9]/_}"
+                s3_target_name="${s3_target_entry,,}"
+                s3_target_name="${s3_target_name//[^a-z0-9]/_}"
+                for s3_target_key in BUCKET CERT_CA_FILE CERT_SKIP_VERIFY EXTRA_OPTS HOST KEY_ID KEY_SECRET PATH PROTOCOL REGION ; do
+                    transform_s3_target_variable "${backup_instance_number}" "${s3_target_id}" "${s3_target_key}" "backup_job_s3_${s3_target_name}_${s3_target_key,,}"
+                done
+            done
+        fi
+
         backup_job_backup_begin=$(echo "${backup_job_backup_begin}" | sed -e "s|'||g" -e 's|"||g')
         if var_true "${DEBUG_BACKUP_INSTANCE_VARIABLE}" ; then cat <<EOF
 ## BEGIN Variable Dump $(TZ=${TIMEZONE} date)
@@ -969,6 +1019,64 @@ dbbackup_check_exit_code() {
     if var_true "${DEBUG_CHECK_EXIT_CODE}" ; then dbbackup_debug off; fi
 }
 
+dbbackup_s3_target_names() {
+    ## Fill the s3_target_names array with the S3 targets configured via S3_TARGETS.
+    ## Without S3_TARGETS the array holds a single empty entry, which makes the callers
+    ## fall back to the single target S3_ variables - the behaviour before this feature.
+    s3_target_names=()
+    if [ -n "${backup_job_s3_targets}" ] ; then
+        local s3_target_entry
+        for s3_target_entry in ${backup_job_s3_targets//,/ } ; do
+            s3_target_entry="${s3_target_entry,,}"
+            s3_target_names+=("${s3_target_entry//[^a-z0-9]/_}")
+        done
+    else
+        s3_target_names=("")
+    fi
+}
+
+dbbackup_s3_target_environment() {
+    ## Prepare the aws cli environment and the s3_target_* variables for a single S3
+    ## target. An empty target name selects the single target S3_ variables.
+    local s3_target="${1}"
+    local s3_target_key
+    local s3_target_var
+
+    for s3_target_key in bucket cert_ca_file cert_skip_verify extra_opts host key_id key_secret path protocol region ; do
+        if [ -n "${s3_target}" ] ; then
+            s3_target_var="backup_job_s3_${s3_target}_${s3_target_key}"
+        else
+            s3_target_var="backup_job_s3_${s3_target_key}"
+        fi
+        printf -v "s3_target_${s3_target_key}" "%s" "${!s3_target_var:-}"
+    done
+
+    if [ -n "${s3_target_key_id}" ] && [ -n "${s3_target_key_secret}" ]; then
+        export AWS_ACCESS_KEY_ID=${s3_target_key_id}
+        export AWS_SECRET_ACCESS_KEY=${s3_target_key_secret}
+    else
+        dbbackup_write_log debug "Variable S3_KEY_ID or S3_KEY_SECRET is not set${s3_target:+ for target '${s3_target}'}. Please ensure sufficiant IAM role is assigned."
+    fi
+    export AWS_DEFAULT_REGION=${s3_target_region}
+
+    s3_ca_cert=""
+    s3_ssl=""
+    PARAM_AWS_ENDPOINT_URL=""
+
+    if [ -f "${s3_target_cert_ca_file}" ] ; then
+        dbbackup_write_log debug "Using Custom CA for S3 Backups"
+        s3_ca_cert="--ca-bundle ${s3_target_cert_ca_file}"
+    fi
+    if var_true "${s3_target_cert_skip_verify}" ; then
+        dbbackup_write_log debug "Skipping SSL verification for HTTPS S3 Hosts"
+        s3_ssl="--no-verify-ssl"
+    fi
+
+    [[ ( -n "${s3_target_host}" ) ]] && PARAM_AWS_ENDPOINT_URL=" --endpoint-url ${s3_target_protocol}://${s3_target_host}"
+
+    return 0
+}
+
 dbbackup_cleanup_old_data() {
     if var_true "${DEBUG_CLEANUP_OLD_DATA}" ; then dbbackup_debug on; fi
     if [ -n "${backup_job_cleanup_time}" ]; then
@@ -998,18 +1106,22 @@ dbbackup_cleanup_old_data() {
                     fi
                 ;;
                 "s3" | "minio" )
-                    dbbackup_write_log info "Cleaning up old backups on S3 storage"
-                    dbbackup_run_as_user aws ${PARAM_AWS_ENDPOINT_URL} s3 ls s3://${backup_job_s3_bucket}/${backup_job_s3_path}/ ${s3_ssl} ${s3_ca_cert} ${backup_job_s3_extra_opts} | grep " DIR " -v | grep " PRE " -v | while read -r s3_file; do
-                        s3_createdate=$(echo $s3_file | awk {'print $1" "$2'})
-                        s3_createdate=$(date -d "$s3_createdate" "+%s")
-                        s3_olderthan=$(echo $(( $(date +%s)-${backup_job_cleanup_time}*60 )))
-                        if [[ $s3_createdate -le $s3_olderthan ]] ; then
-                            s3_filename=$(echo $s3_file | awk {'print $4'})
-                            if [ "$s3_filename" != "" ] ; then
-                                dbbackup_write_log debug "Deleting $s3_filename"
-                                dbbackup_run_as_user aws ${PARAM_AWS_ENDPOINT_URL} s3 rm s3://${backup_job_s3_bucket}/${backup_job_s3_path}/${s3_filename} ${s3_ssl} ${s3_ca_cert} ${backup_job_s3_extra_opts}
+                    dbbackup_s3_target_names
+                    for s3_target in "${s3_target_names[@]}" ; do
+                        dbbackup_write_log info "Cleaning up old backups on S3 storage${s3_target:+ - target '${s3_target}'}"
+                        dbbackup_s3_target_environment "${s3_target}"
+                        dbbackup_run_as_user aws ${PARAM_AWS_ENDPOINT_URL} s3 ls s3://${s3_target_bucket}/${s3_target_path}/ ${s3_ssl} ${s3_ca_cert} ${s3_target_extra_opts} | grep " DIR " -v | grep " PRE " -v | while read -r s3_file; do
+                            s3_createdate=$(echo $s3_file | awk {'print $1" "$2'})
+                            s3_createdate=$(date -d "$s3_createdate" "+%s")
+                            s3_olderthan=$(echo $(( $(date +%s)-${backup_job_cleanup_time}*60 )))
+                            if [[ $s3_createdate -le $s3_olderthan ]] ; then
+                                s3_filename=$(echo $s3_file | awk {'print $4'})
+                                if [ "$s3_filename" != "" ] ; then
+                                    dbbackup_write_log debug "Deleting $s3_filename"
+                                    dbbackup_run_as_user aws ${PARAM_AWS_ENDPOINT_URL} s3 rm s3://${s3_target_bucket}/${s3_target_path}/${s3_filename} ${s3_ssl} ${s3_ca_cert} ${s3_target_extra_opts}
+                                fi
                             fi
-                        fi
+                        done
                     done
                 ;;
             esac
@@ -1512,29 +1624,30 @@ EOF
                 fi
             ;;
             "s3" | "minio" )
-                dbbackup_write_log debug "Moving backup to S3 Bucket"
-                if [ -n "${backup_job_s3_key_id}" ] && [ -n "${backup_job_s3_key_secret}" ]; then
-                    export AWS_ACCESS_KEY_ID=${backup_job_s3_key_id}
-                    export AWS_SECRET_ACCESS_KEY=${backup_job_s3_key_secret}
-                else
-                    dbbackup_write_log debug "Variable S3_KEY_ID or S3_KEY_SECRET is not set. Please ensure sufficiant IAM role is assigned."
-                fi
-                export AWS_DEFAULT_REGION=${backup_job_s3_region}
-                if [ -f "${backup_job_s3_cert_ca_file}" ] ; then
-                    dbbackup_write_log debug "Using Custom CA for S3 Backups"
-                    s3_ca_cert="--ca-bundle ${backup_job_s3_cert_ca_file}"
-                fi
-                if var_true "${backup_job_s3_cert_skip_verify}" ; then
-                    dbbackup_write_log debug "Skipping SSL verification for HTTPS S3 Hosts"
-                    s3_ssl="--no-verify-ssl"
-                fi
+                dbbackup_s3_target_names
+                move_exit_code=0
+                for s3_target in "${s3_target_names[@]}" ; do
+                    dbbackup_write_log debug "Moving backup to S3 Bucket${s3_target:+ - target '${s3_target}'}"
+                    dbbackup_s3_target_environment "${s3_target}"
+
+                    silent aws ${PARAM_AWS_ENDPOINT_URL} s3 cp ${temporary_directory}/${backup_job_filename} s3://${s3_target_bucket}/${s3_target_path}/${backup_job_filename} ${s3_ssl} ${s3_ca_cert} ${s3_target_extra_opts}
+                    s3_target_exit_code=$?
+                    if [ "${s3_target_exit_code}" != "0" ] ; then
+                        dbbackup_write_log error "Failed moving backup to S3 Bucket${s3_target:+ - target '${s3_target}'} - exit code ${s3_target_exit_code}"
+                        move_exit_code=${s3_target_exit_code}
+                    fi
 
-                [[ ( -n "${backup_job_s3_host}" ) ]] && PARAM_AWS_ENDPOINT_URL=" --endpoint-url ${backup_job_s3_protocol}://${backup_job_s3_host}"
+                    if [ "${backup_job_checksum}" != "none" ] ; then
+                        silent dbbackup_run_as_user ${play_fair} aws ${PARAM_AWS_ENDPOINT_URL} s3 cp ${temporary_directory}/*.${checksum_extension} s3://${s3_target_bucket}/${s3_target_path}/ ${s3_ssl} ${s3_ca_cert} ${s3_target_extra_opts}
+                        s3_target_exit_code=$?
+                        if [ "${s3_target_exit_code}" != "0" ] ; then
+                            dbbackup_write_log error "Failed moving checksum to S3 Bucket${s3_target:+ - target '${s3_target}'} - exit code ${s3_target_exit_code}"
+                            move_exit_code=${s3_target_exit_code}
+                        fi
+                    fi
+                done
 
-                silent aws ${PARAM_AWS_ENDPOINT_URL} s3 cp ${temporary_directory}/${backup_job_filename} s3://${backup_job_s3_bucket}/${backup_job_s3_path}/${backup_job_filename} ${s3_ssl} ${s3_ca_cert} ${backup_job_s3_extra_opts}
-                move_exit_code=$?
                 if [ "${backup_job_checksum}" != "none" ] ; then
-                    silent dbbackup_run_as_user ${play_fair} aws ${PARAM_AWS_ENDPOINT_URL} s3 cp ${temporary_directory}/*.${checksum_extension} s3://${backup_job_s3_bucket}/${backup_job_s3_path}/ ${s3_ssl} ${s3_ca_cert} ${backup_job_s3_extra_opts}
                     dbbackup_run_as_user rm -rf "${temporary_directory}"/"${backup_job_filename}"."${checksum_extension}"
                 fi