fix(charm): recover temp tablespace after cross-revision restore#1874
Draft
marceloneppel wants to merge 2 commits into
Draft
fix(charm): recover temp tablespace after cross-revision restore#1874marceloneppel wants to merge 2 commits into
marceloneppel wants to merge 2 commits into
Conversation
A backup taken on a pre-versioned-storage revision restores the temp tablespace at the old non-versioned path. Install and refresh already migrate it to the versioned path, but restore did not — so after a cross-revision restore the tablespace stayed at the old location. A new replica's pg_basebackup then streams into the charm-created versioned directory nested under it, hitting a non-empty target and failing, and pgBackRest rejects the nested tablespace links with error [070]. Run the existing temp-tablespace migration in the restore-success path, before clearing the restoring-backup flags, so the restored primary is on the versioned layout before any replica streams from it or a backup runs. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
On a tmpfs reboot the temp tablespace directory is recreated, and set_up_database renames the existing temp tablespace to a timestamped leftover (temp_<timestamp>) whenever the directory's mode differs from the expected storage permissions. _ensure_storage_layout created the directory without setting its mode, so the default umask left it at a mode that never matched, triggering the rename on every reboot and accumulating leftover tablespaces the operator has to clean up. Set the directory mode to the expected storage permissions on creation so the rename-and-leave path is not taken. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
github-actions
Bot
added
the
Libraries: Out of sync
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Restoring a backup taken on a pre-versioned-storage revision (before the
16/mainsubdirectory layout) into a current revision leaves the
temptablespace at the oldnon-versioned path. Install and refresh already migrate it to the versioned path, but the
restore-success path did not — so after a cross-revision restore the tablespace stayed at the
old location, nested under the charm-created versioned directory. A new replica's
pg_basebackupthen streamed into that versioned directory and failed withdirectory "…" is not empty, andcreate-backupfailed with pgBackRest error[070] … destination is a subdirectory of link ….Separately, on a tmpfs reboot the temp tablespace directory is recreated without its expected
mode, so
set_up_databasetook the rename-and-leave path (ALTER TABLESPACE temp RENAME TO temp_<timestamp>) on every reboot, accumulating leftover tablespaces — which is what producedthe nested-link state above on an affected cluster.
Solution
_was_restore_successful, before clearing therestoring-backup flags, so the restored primary is on the versioned layout before any replica
streams from it or a backup runs. The check defers (preserving the flags) until the migration
completes.
POSTGRESQL_STORAGE_PERMISSIONSon creation in_ensure_storage_layout, soset_up_database'spermissions_need_fixgate is not tripped onreboot and the rename-and-leave path is not taken.
Validated end-to-end with
tox run -e unit, and by reproducing the real cross-revision restoreon a cluster: the charm now migrates the tablespace automatically on restore, and
create-backupand scale-up succeed without manual intervention.
Checklist