Report a failed move to the backup location as a failed backup to Zabbix - #493
Open
cfoellmann wants to merge 1 commit into
Open
Report a failed move to the backup location as a failed backup to Zabbix#493cfoellmann wants to merge 1 commit into
cfoellmann wants to merge 1 commit into
Conversation
dbbackup.backup.status carried ${exit_code}, which is the exit code of the dump
alone - it is assigned right after the dump command and before the backup is
moved to its destination. A dump that was written correctly but never reached
its S3 bucket, filesystem path or blob container was therefore reported to
Zabbix as a successful backup, and the template's only status trigger,
last(/DB Backup4/dbbackup.backup.status.[{#NAME}])=1, stayed silent.
The remaining items do not catch it either: size, filename and especially
datetime are read from the dump in the temporary directory before the move, so
even an age based fuzzytime() trigger sees a fresh backup while the destination
holds nothing. Outside of Zabbix the failure is only visible in the container
log and through dbbackup_notify, which does nothing unless NOTIFICATION_TYPE is
configured - so a destination that silently stops accepting uploads can go
unnoticed for a long time.
Send the move result along with the dump result instead. The status stays 0
only when both the dump and the move succeeded, so the existing trigger fires
without any change to the template.
Verified with the container image against a stubbed zabbix_sender: an
unreachable S3 endpoint now reports status 1 where the unpatched script
reported 0, and a successful backup still reports 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
dbbackup.backup.statuscarries${exit_code}, which is the exit code of the dump alone — it is assigned right after the dump command and beforedbbackup_move_dbbackupruns:A dump that was written correctly but never reached its S3 bucket, filesystem path or blob container is therefore reported to Zabbix as a successful backup, and the template's only status trigger,
last(/DB Backup4/dbbackup.backup.status.[{#NAME}])=1, stays silent.The remaining items do not catch it either:
dbbackup_size,filenameand especiallydbbackup_dateare read from the dump in the temporary directory before the move, so even an age basedfuzzytime()trigger sees a fresh backup while the destination holds nothing.Outside of Zabbix the failure is visible only in the container log and through
dbbackup_notify, which does nothing unlessNOTIFICATION_TYPEis configured. A destination that silently stops accepting uploads can therefore go unnoticed for a long time.The change
Send the move result along with the dump result. The status stays
0only when both the dump and the move succeeded, so the existing trigger fires without any change to the template:This changes behaviour for existing installations: jobs whose upload currently fails silently will start showing up as
Backup - Failed with errors. That is the intent — but it is worth calling out, since it may surface failures that were previously invisible.Testing
Verified with the released container image against a stubbed
zabbix_sender, three runs:dbbackup.backup.status01main0The third run is the control: same failure, without the patch.
Note
This branch is independent of my two other PRs. It adds a
4.9.3CHANGELOG heading, as does #494 — whichever merges second will need a trivial conflict resolution at the top of the file.