Skip to content

Report a failed move to the backup location as a failed backup to Zabbix - #493

Open
cfoellmann wants to merge 1 commit into
nfrastack:mainfrom
cfoellmann:fix/zabbix-status-move-exit-code
Open

Report a failed move to the backup location as a failed backup to Zabbix#493
cfoellmann wants to merge 1 commit into
nfrastack:mainfrom
cfoellmann:fix/zabbix-status-move-exit-code

Conversation

@cfoellmann

@cfoellmann cfoellmann commented Aug 30, 2026

Copy link
Copy Markdown

The problem

dbbackup.backup.status carries ${exit_code}, which is the exit code of the dump alone — it is assigned right after the dump command and before dbbackup_move_dbbackup runs:

silent dbbackup_run_as_user ${play_fair} mongodump ...
exit_code=$?
...
dbbackup_move_dbbackup          # sets move_exit_code
dbbackup_check_exit_code move "${backup_job_filename}"
dbbackup_post_dbbackup "${backup_job_db_name}"   # sends ${exit_code} to Zabbix

A dump that was written correctly but never reached its S3 bucket, filesystem path or blob container is therefore reported to Zabbix as a successful backup, and the template's only status trigger, last(/DB Backup4/dbbackup.backup.status.[{#NAME}])=1, stays silent.

The remaining items do not catch it either: dbbackup_size, filename and especially dbbackup_date are read from the dump in the temporary directory before the move, so even an age based fuzzytime() trigger sees a fresh backup while the destination holds nothing.

Outside of Zabbix the failure is visible only in the container log and through dbbackup_notify, which does nothing unless NOTIFICATION_TYPE is configured. A destination that silently stops accepting uploads can therefore go unnoticed for a long time.

The change

Send the move result along with the dump result. The status stays 0 only when both the dump and the move succeeded, so the existing trigger fires without any change to the template:

local zabbix_status="${exit_code}"
if [ "${move_exit_code:-0}" != "0" ] ; then zabbix_status=1 ; fi

This changes behaviour for existing installations: jobs whose upload currently fails silently will start showing up as Backup - Failed with errors. That is the intent — but it is worth calling out, since it may surface failures that were previously invisible.

Testing

Verified with the released container image against a stubbed zabbix_sender, three runs:

Run Script dbbackup.backup.status
Move succeeded (filesystem) patched 0
S3 endpoint unreachable patched 1
S3 endpoint unreachable unpatched main 0

The third run is the control: same failure, without the patch.

Note

This branch is independent of my two other PRs. It adds a 4.9.3 CHANGELOG heading, as does #494 — whichever merges second will need a trivial conflict resolution at the top of the file.

dbbackup.backup.status carried ${exit_code}, which is the exit code of the dump
alone - it is assigned right after the dump command and before the backup is
moved to its destination. A dump that was written correctly but never reached
its S3 bucket, filesystem path or blob container was therefore reported to
Zabbix as a successful backup, and the template's only status trigger,
last(/DB Backup4/dbbackup.backup.status.[{#NAME}])=1, stayed silent.

The remaining items do not catch it either: size, filename and especially
datetime are read from the dump in the temporary directory before the move, so
even an age based fuzzytime() trigger sees a fresh backup while the destination
holds nothing. Outside of Zabbix the failure is only visible in the container
log and through dbbackup_notify, which does nothing unless NOTIFICATION_TYPE is
configured - so a destination that silently stops accepting uploads can go
unnoticed for a long time.

Send the move result along with the dump result instead. The status stays 0
only when both the dump and the move succeeded, so the existing trigger fires
without any change to the template.

Verified with the container image against a stubbed zabbix_sender: an
unreachable S3 endpoint now reports status 1 where the unpatched script
reported 0, and a successful backup still reports 0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant