From 78c8a9e14c4232a007643748b29c827c6aa5778f Mon Sep 17 00:00:00 2001 From: Optic00 Date: Sat, 29 Aug 2026 10:58:06 +0200 Subject: [PATCH 1/2] feat: add Docker backup and restore workflow --- .github/workflows/backup-contract.yml | 21 + README.md | 1 + deploy/backup-restore.md | 188 ++++++++ deploy/backup.sh | 655 ++++++++++++++++++++++++++ deploy/docker-compose-main.yml | 1 + deploy/test-backup.sh | 561 ++++++++++++++++++++++ 6 files changed, 1427 insertions(+) create mode 100644 .github/workflows/backup-contract.yml create mode 100644 deploy/backup-restore.md create mode 100755 deploy/backup.sh create mode 100755 deploy/test-backup.sh diff --git a/.github/workflows/backup-contract.yml b/.github/workflows/backup-contract.yml new file mode 100644 index 000000000..3eb378968 --- /dev/null +++ b/.github/workflows/backup-contract.yml @@ -0,0 +1,21 @@ +name: Backup Contract + +on: + push: + branches: [main] + pull_request: + branches: [main] + +permissions: + contents: read + +jobs: + backup-contract: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + - name: Run Docker-free backup contract test + run: | + bash -n deploy/backup.sh + bash -n deploy/test-backup.sh + bash deploy/test-backup.sh diff --git a/README.md b/README.md index 11da542a2..169687b3a 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,7 @@ We are especially interested in early bug reports and real-world feedback about - [Build instructions](BUILD.md) - [Contributing guide](CONTRIBUTING.md) - [Logging configuration](LOGGING.md) +- [Docker backup and restore](deploy/backup-restore.md) - [Product documentation](https://windshift.sh/docs) ## License diff --git a/deploy/backup-restore.md b/deploy/backup-restore.md new file mode 100644 index 000000000..c3bd1ee38 --- /dev/null +++ b/deploy/backup-restore.md @@ -0,0 +1,188 @@ +# Backing up and restoring Windshift + +`backup.sh` is a conservative host-side workflow for the stock Docker Compose deployments in this repository. +It is not a general-purpose Docker or database recovery tool. +It backs up the complete configured `/data` tree and, for the supported PostgreSQL shape, a custom-format logical dump. + +The helper image is pinned to the Alpine digest used by `deploy/coding-agent/Dockerfile`. +It has no network, a read-only root filesystem, and no-new-privileges. +The backup helper receives Windshift volumes read-only. +The restore helper receives a writable data volume only after all preflight checks and a local rollback snapshot have completed. + +## Supported scope + +The workflow requires exactly one Compose `windshift` container using the stock `/windshift` entrypoint and exactly one persistent mount at the configured data path. +Additional mounts below that path are rejected because deleting a mounted child can leave both restore and rollback incomplete. +The backup directory and rollback root must be outside that mount's host source so an archive cannot include or delete itself. +Replicas are rejected. +It derives a stable, domain-separated hash from the Compose project and service labels and uses a system-wide mode-700 lock directory under `/var/tmp`. + +SQLite is supported only when `DB_PATH` is below `/data`. +An empty `DB_PATH` is rejected because Windshift would otherwise fall back to a path outside the mounted data tree. +The database path is recorded in the manifest and must exist as a nonempty regular archive entry. +The script also checks `ATTACHMENT_PATH`, `PLUGIN_DIR`, `AI_PROMPTS_DIR`, `LLM_PROVIDERS_FILE`, and trimmed comma-separated `PLUGIN_DIRS` when configured. +An empty `PLUGIN_DIR` is rejected because Windshift would otherwise fall back to a relative path outside the data mount. +SSH host keys and SSH-enabled deployments are outside this workflow because the image-relative default path is not guaranteed to be under `/data`. +Any extra host mounts or persisted paths outside `/data` are out of scope and cause a safe failure when they are configured through these variables. + +PostgreSQL is supported only when all of the following are true: + +- `DB_TYPE=postgres` and `POSTGRES_CONNECTION_STRING` is unset. +- `POSTGRES_HOST` exactly equals the selected Compose service. +- `POSTGRES_PORT` is unset or 5432. +- `POSTGRES_USER` and `POSTGRES_DB` exactly match that service's container environment. +- Exactly one PostgreSQL service container exists. + +The automatic service candidates are `postgres` and `db`. +Use `--postgres-service NAME` for another stock-style service. +On restore, that explicit option overrides the manifest value. +External databases, connection strings, non-default ports, custom database selection, and replicas are deliberately unsupported. +Any command-line use of `-postgres-connection-string`, `--postgres-connection-string`, `-pg-conn`, or `--pg-conn` is also out of scope and fails closed. +The path override flags `-db`, `--db`, `-attachment-path`, `--attachment-path`, `-llm-providers`, `--llm-providers`, `-ai-prompts-dir`, and `--ai-prompts-dir` are out of scope too. +Environment values containing CR or LF are rejected because this workflow cannot safely fingerprint or archive them. +Use a database-specific procedure for those deployments. + +Use the same PostgreSQL major version for backup and restore. +Restore only into the same Windshift image and schema generation that produced the backup unless a separately verified migration plan says otherwise. + +## Create a backup + +Run from the Compose directory: + + cd deploy + + ./backup.sh backup /srv/windshift-backups/windshift-$(date -u +%Y%m%dT%H%M%SZ) + +The backup path must be an absolute, normalized host path without a colon because it is passed to Docker's `-v` syntax. +The created directory is mode 700. +It contains `data.tar.gz`, `manifest.env`, and `checksums.sha256`. +PostgreSQL backups also contain `database.dump`. + +The script takes an atomic lock before stopping the app or creating backup files. +If a lock exists, do not run another operation. +First confirm that no backup or restore is running, then remove only the reported stale lock directory. + +SHA-256 checksums detect accidental corruption and incomplete copies. +They do not authenticate a backup. +Use access-controlled, authenticated, encrypted storage and transport. + +The manifest stores a domain-separated SHA-256 fingerprint of the effective `SSO_SECRET`, or `SESSION_SECRET` when no `SSO_SECRET` is present. +This is a compatibility check, not a password verifier. +Weak secrets therefore still produce easily guessable fingerprints. + +Pass `--include-sso-secret` only for an explicitly encrypted secret-recovery copy. +It writes the raw effective secret as mode-600 `sso-secret` inside the mode-700 backup and includes it in the checksum file. +This artifact is highly sensitive and must never be logged or copied to ordinary backup media. + +## Restore + +Run from the same Compose directory: + + cd deploy + + ./backup.sh restore --force /srv/windshift-backups/windshift-20260829T120000Z + +Before stopping Windshift, restore rejects symlinked or non-regular input artifacts, copies approved artifacts into a private staging directory, and verifies only that staged snapshot thereafter. +The helper checks the staged archive in the same image that performs extraction. +It accepts only regular files and directories, rejects absolute and parent paths, and rejects symbolic links, hard links, devices, and FIFOs. + +Restore compares the live secret fingerprint before any state change. +On mismatch it fails closed and never writes a secret into Compose. +Configure the source secret in Compose or the secret manager first. + +After the app stops, the script creates a mode-700 rollback directory under `--rollback-root` or `WINDSHIFT_BACKUP_ROLLBACK_ROOT`. +The default is disk-backed `/var/tmp`. +That directory must already exist and be writable. +Choose a local, private filesystem with space for a full compressed `/data` copy, staging copy, and, for PostgreSQL, a second logical dump. +Do not put it on volatile `/tmp` unless its loss during reboot is acceptable. + +For PostgreSQL the rollback contains both `/data` and a pre-restore database dump. +The script writes SHA-256 checksums for retained rollback artifacts and structurally validates custom-format PostgreSQL dumps with `pg_restore --list` before relying on them. +The target database restore uses `pg_restore --single-transaction --exit-on-error`. +It cleans and restores archive objects, not necessarily unrelated database objects. + +The rollback is retained until the started application is healthy. +When Docker exposes a health check, the script waits for it for up to `WINDSHIFT_BACKUP_HEALTH_TIMEOUT` seconds, default 90. +Without a Docker health status, it executes `/windshift healthcheck` through Compose until the configured deadline. +Increase `WINDSHIFT_BACKUP_HEALTH_TIMEOUT` before restore when startup or migrations can legitimately take longer than 90 seconds. +If the target is unhealthy, it restores both rollback data and PostgreSQL before trying the old state. +If PostgreSQL state is uncertain because of an interruption or failed database command, Windshift remains stopped and the error retains both rollback and staged paths. +The recovery lock also remains in place because a detached server-side `pg_restore` may still be running. +Remove that lock only after confirming that no backup, restore, or PostgreSQL restore process is active and after resolving the database state. +If Windshift was already stopped before restore, the script cannot perform a health check or automatic start. +It reports success with the retained rollback path so the operator can validate a manual start before removing it. + +The command uses a 60-second Compose stop timeout. +An interrupted command, host reboot, or Docker failure can still leave the service stopped. +Read the printed recovery paths before restarting it. +On a confirmed healthy restore, cleanup failures only warn and retain the private rollback or stage instead of changing the restored state. +The ephemeral backup helper streams its tar archive to a host-created mode-600 file, so it does not need to chown host files. +The Docker-free contract test does not prove behavior against a real Docker daemon, image, filesystem driver, or PostgreSQL server. + +## Manual emergency recovery + +This is a destructive emergency procedure. +Leave Windshift stopped and first copy every reported stage and rollback directory to immutable, access-controlled storage. +Use only the same Windshift image/schema generation and the same PostgreSQL major version. + +Before stopping or deleting anything, validate the retained checksums and data archive in a separate preflight. +Use the SHA-256 tool available on the host and the exact pinned helper image shown below: + + cd deploy + (cd "$ROLLBACK_DIR" && sha256sum -c checksums.sha256) + docker run --rm --network none --read-only --security-opt no-new-privileges -v "$ROLLBACK_DIR:/backup:ro" alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d sh -ec ' + test -s /backup/data.tar.gz + tar -tzf /backup/data.tar.gz >/dev/null + tar -tvzf /backup/data.tar.gz >/dev/null + tar -tzf /backup/data.tar.gz | awk '\''{ + path = $0 + if (path == "./") key = "." + else { + if (substr(path, 1, 2) != "./") exit 1 + path = substr(path, 3) + if (path == "" || path ~ /(^|\/)\.($|\/)/ || path ~ /(^|\/)\.\.($|\/)/ || path ~ /\/\//) exit 1 + sub(/\/$/, "", path) + if (path == "") exit 1 + key = path + } + if (seen[key]++) exit 1 + count++ + } END { if (!count) exit 1 }'\'' + if tar -tvzf /backup/data.tar.gz | grep -Eqv "^[-d]"; then exit 1; fi + if tar -tvzf /backup/data.tar.gz | grep -E "^-.* -> "; then exit 1; fi + ' + +If the host provides `shasum` instead, replace the checksum command with `shasum -a 256 -c checksums.sha256`. +Do not continue unless every command succeeds. + +For PostgreSQL, validate the retained database dump before changing `/data`. +Set `POSTGRES_SERVICE` to the confirmed service name and keep `remote_dump` for the restore step below: + + POSTGRES_SERVICE=postgres + db_id=$(docker compose ps --all -q "$POSTGRES_SERVICE") + [ "$(printf '%s\n' "$db_id" | sed '/^$/d' | wc -l | tr -d ' ')" = 1 ] || exit 1 + remote_dump=/tmp/windshift-manual-rollback-$$.dump + docker cp "$ROLLBACK_DIR/database-before.dump" "$db_id:$remote_dump" + docker compose exec -T "$POSTGRES_SERVICE" sh -ec 'pg_restore --list "$1" >/dev/null' sh "$remote_dump" + +For `/data`, resolve exactly one Windshift container before stopping it: + + cd deploy + windshift_id=$(docker compose ps --all -q windshift) + [ "$(printf '%s\n' "$windshift_id" | sed '/^$/d' | wc -l | tr -d ' ')" = 1 ] || exit 1 + mount_destinations=$(docker inspect -f '{{range .Mounts}}{{println .Destination}}{{end}}' "$windshift_id") + [ "$(printf '%s\n' "$mount_destinations" | grep -Fxc /data)" = 1 ] || exit 1 + if printf '%s\n' "$mount_destinations" | grep -Eq '^/data/'; then exit 1; fi + docker compose stop --timeout 60 windshift || exit 1 + [ "$(docker compose ps --all -q windshift)" = "$windshift_id" ] || exit 1 + [ "$(docker inspect -f '{{.State.Running}}' "$windshift_id")" = false ] || exit 1 + docker run --rm --network none --read-only --security-opt no-new-privileges --volumes-from "$windshift_id" -v "$ROLLBACK_DIR:/backup:ro" alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d sh -ec 'find /data -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +; tar -C /data -xzf /backup/data.tar.gz' + +For PostgreSQL, restore the already validated remote dump only after confirming the target database again: + + docker compose exec -T "$POSTGRES_SERVICE" sh -ec 'pg_restore --clean --if-exists --no-owner --no-privileges --single-transaction --exit-on-error -U "$POSTGRES_USER" -d "$POSTGRES_DB" "$1"' sh "$remote_dump" + docker compose exec -T "$POSTGRES_SERVICE" rm -f "$remote_dump" + +The PostgreSQL rollback cleans objects listed in its dump but may leave unrelated objects created by a newer schema. +Remove any container path printed as `remote=` after the database state has been recovered and the retained dump has been preserved elsewhere. +Only then start Windshift and run its health check. diff --git a/deploy/backup.sh b/deploy/backup.sh new file mode 100755 index 000000000..0f022b3c4 --- /dev/null +++ b/deploy/backup.sh @@ -0,0 +1,655 @@ +#!/usr/bin/env bash +set -euo pipefail + +PROGRAM=${0##*/} +readonly FORMAT_VERSION=3 SERVICE=windshift +readonly DEFAULT_HELPER_IMAGE='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' +COMPOSE=(docker compose) +DATA_PATH=/data POSTGRES_SERVICE= HELPER_IMAGE=${WINDSHIFT_BACKUP_HELPER_IMAGE:-$DEFAULT_HELPER_IMAGE} +ROLLBACK_ROOT=${WINDSHIFT_BACKUP_ROLLBACK_ROOT:-/var/tmp} +INCLUDE_SSO_SECRET=false ACTION= FORCE=false BACKUP_PATH= +BACKUP_DEST_CREATED=false BACKUP_READY=false +WINDSHIFT_CONTAINER= DATA_MOUNT_SOURCE= WAS_RUNNING=false LOCK_DIR= LOCK_PHYSICAL= STAGE_DIR= ROLLBACK_DIR= ROLLBACK_CHECKSUM= +DB_ROLLBACK= DB_ROLLBACK_CHECKSUM= PHASE=preflight CLEANUP_RUNNING=false SQLITE_DB_REL= LIVE_TYPE= PG_SERVICE= result= +HEALTH_TIMEOUT=${WINDSHIFT_BACKUP_HEALTH_TIMEOUT:-90} PG_REMOTE= PG_REMOTE_SERVICE= + +die() { printf '%s: %s\n' "$PROGRAM" "$*" >&2; exit 1; } +warn() { printf '%s: warning: %s\n' "$PROGRAM" "$*" >&2; } +usage() { cat </dev/null 2>&1 || die "required command not found: $1"; } +sha256() { + if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | awk '{print $1}' + else shasum -a 256 "$1" | awk '{print $1}'; fi +} +sha256_stdin() { + if command -v sha256sum >/dev/null 2>&1; then sha256sum | awk '{print $1}' + else shasum -a 256 | awk '{print $1}'; fi +} +valid_container_path() { + case "$1" in /*) ;; *) return 1;; esac + case "$1/" in *'/../'*|*'/./'*|*'//'*) return 1;; esac + case "$1" in *$'\n'*|*$'\r'*|*$'\t'*) return 1;; esac +} +normalise_container_path() { + local p=$1 + while [ "$p" != / ] && [ "${p%/}" != "$p" ]; do p=${p%/}; done + valid_container_path "$p" || return 1 + [ "$p" != / ] || return 1 + printf '%s\n' "$p" +} +normalise_host_path() { + local p=$1 + case "$p" in /*) ;; *) return 1;; esac + while [ "$p" != / ] && [ "${p%/}" != "$p" ]; do p=${p%/}; done + case "$p" in *:*) return 1;; esac + case "/${p#/}/" in *'/./'*|*'/../'*|*'//'*) return 1;; esac + case "$p" in *$'\n'*|*$'\r'*|*$'\t'*) return 1;; esac + printf '%s\n' "$p" +} +physical_existing_dir() { (cd -P -- "$1" 2>/dev/null && pwd -P); } +physical_new_child_path() { + local path=$1 parent name physical_parent + parent=${path%/*}; name=${path##*/} + [ -n "$parent" ] || parent=/ + [ -n "$name" ] || return 1 + physical_parent=$(physical_existing_dir "$parent") || return 1 + printf '%s/%s\n' "${physical_parent%/}" "$name" +} +env_value() { + local entry key=$2 value= found=false sentinel=false + while IFS= read -r -d '' entry; do + if [ "$entry" = __WINDSHIFT_ENV_END__ ]; then sentinel=true; continue; fi + case "$entry" in "$key"=*) [ "$found" = false ] || return 1; value=${entry#*=}; found=true;; esac + done < <(docker inspect -f '{{range .Config.Env}}{{printf "%s\x00" .}}{{end}}{{printf "__WINDSHIFT_ENV_END__\x00"}}' "$1") || return 1 + [ "$sentinel" = true ] || return 1 + case "$value" in *$'\r'*|*$'\n'*) return 1;; esac + printf '%s' "$value" +} +read_env() { + local value + value=$(env_value "$1" "$2") || return 1 + printf '%s' "$value" +} +container() { + local ids count + ids=$("${COMPOSE[@]}" ps --all -q "$1") || return 1 + count=$(printf '%s\n' "$ids" | sed '/^$/d' | wc -l | tr -d ' ') || return 1 + [ "$count" = 1 ] || return 1 + printf '%s\n' "$ids" +} +require_container() { + local id + id=$(container "$1") || die "expected exactly one Compose container for '$1'; replicas and missing services are unsupported" + printf '%s\n' "$id" +} +container_running_state() { + local state + state=$(docker inspect -f '{{.State.Running}}' "$1") || return 1 + case "$state" in true|false) printf '%s\n' "$state";; *) return 1;; esac +} +running() { [ "$(container_running_state "$1")" = true ]; } +stopped() { [ "$(container_running_state "$1")" = false ]; } +health_status() { docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{end}}' "$1"; } +data_mount_present() { + local field destination= expect=destination count=0 nested=false sentinel=false + DATA_MOUNT_SOURCE= + while IFS= read -r -d '' field; do + if [ "$field" = __WINDSHIFT_MOUNT_END__ ]; then sentinel=true; continue; fi + if [ "$expect" = destination ]; then + destination=$field + expect=source + else + if [ "$destination" = "$DATA_PATH" ]; then count=$((count + 1)); DATA_MOUNT_SOURCE=$field + else case "$destination" in "$DATA_PATH"/*) nested=true;; esac + fi + expect=destination + fi + done < <(docker inspect -f '{{range .Mounts}}{{printf "%s\x00%s\x00" .Destination .Source}}{{end}}{{printf "__WINDSHIFT_MOUNT_END__\x00"}}' "$WINDSHIFT_CONTAINER") || return 1 + [ "$sentinel" = true ] && [ "$expect" = destination ] && [ "$count" = 1 ] && [ "$nested" = false ] || return 1 + DATA_MOUNT_SOURCE=$(normalise_host_path "$DATA_MOUNT_SOURCE") || return 1 + if [ -d "$DATA_MOUNT_SOURCE" ]; then DATA_MOUNT_SOURCE=$(physical_existing_dir "$DATA_MOUNT_SOURCE") || return 1; fi + [ "$DATA_MOUNT_SOURCE" != / ] +} +require_outside_data_mount() { + case "$1" in "$DATA_MOUNT_SOURCE"|"$DATA_MOUNT_SOURCE"/*) die "$2 must be outside the host source of --data-path ($DATA_MOUNT_SOURCE)";; esac +} +inside_data() { + local value=$1 name=$2 normal + [ -z "$value" ] && return 0 + normal=$(normalise_container_path "$value") || die "$name is not a safe absolute container path" + case "$normal" in "$DATA_PATH"|"$DATA_PATH"/*) ;; *) die "$name ($value) is outside --data-path ($DATA_PATH)";; esac +} +trim() { sed 's/^[[:space:]]*//;s/[[:space:]]*$//' <<<"$1"; } +data_relative_path() { + local path=$1 rel + path=$(normalise_container_path "$path") || return 1 + inside_data "$path" DB_PATH + [ "$path" != "$DATA_PATH" ] || return 1 + rel=${path#"$DATA_PATH"/} + case "$rel" in ''|/*|*'//'|*'..'*|*[!A-Za-z0-9._/-]*) return 1;; esac + printf '%s\n' "$rel" +} +effective_secret() { + local sso session + sso=$(read_env "$WINDSHIFT_CONTAINER" SSO_SECRET) || return 1 + session=$(read_env "$WINDSHIFT_CONTAINER" SESSION_SECRET) || return 1 + if [ -n "$sso" ]; then printf '%s' "$sso" + elif [ -n "$session" ]; then printf '%s' "$session" + else return 1; fi +} +secret_fingerprint() { + local secret + secret=$(effective_secret) || return 1 + printf 'windshift-backup-sso-secret-v1\0%s' "$secret" | sha256_stdin +} +database_type() { + local db_type connection entrypoint args ssh_enabled token + db_type=$(read_env "$WINDSHIFT_CONTAINER" DB_TYPE) || return 1 + connection=$(read_env "$WINDSHIFT_CONTAINER" POSTGRES_CONNECTION_STRING) || return 1 + ssh_enabled=$(read_env "$WINDSHIFT_CONTAINER" SSH_ENABLED) || return 1 + [ -z "$connection" ] || return 2 + case "$ssh_enabled" in true|TRUE|True|1|yes|YES|Yes) return 3;; esac + entrypoint=$(docker inspect -f '{{range .Config.Entrypoint}}{{println .}}{{end}}' "$WINDSHIFT_CONTAINER") || return 1 + [ "$entrypoint" = /windshift ] || return 3 + args=$(docker inspect -f '{{range .Config.Cmd}}{{println .}}{{end}}' "$WINDSHIFT_CONTAINER") || return 1 + while IFS= read -r token; do + case "$token" in + -postgres-connection-string|--postgres-connection-string|-postgres-connection-string=*|--postgres-connection-string=*|-pg-conn|--pg-conn|-pg-conn=*|--pg-conn=*|-ssh|--ssh|-ssh=*|--ssh=*|-db|--db|-db=*|--db=*|-attachment-path|--attachment-path|-attachment-path=*|--attachment-path=*|-llm-providers|--llm-providers|-llm-providers=*|--llm-providers=*|-ai-prompts-dir|--ai-prompts-dir|-ai-prompts-dir=*|--ai-prompts-dir=*) return 3;; + esac + done <<<"$args" + case "$db_type" in ''|sqlite) printf sqlite;; postgres) printf postgres;; *) return 3;; esac +} +check_paths() { + local type=$1 value path sqlite_path + if [ "$type" = sqlite ]; then + value=$(read_env "$WINDSHIFT_CONTAINER" DB_PATH) || return 1 + [ -n "$value" ] || return 1 + sqlite_path=$value + SQLITE_DB_REL=$(data_relative_path "$sqlite_path") || return 1 + fi + for path in ATTACHMENT_PATH PLUGIN_DIR AI_PROMPTS_DIR LLM_PROVIDERS_FILE; do + value=$(read_env "$WINDSHIFT_CONTAINER" "$path") || return 1 + case "$path" in + PLUGIN_DIR) [ -n "$value" ] || return 1; inside_data "$value" "$path";; + *) [ -z "$value" ] || inside_data "$value" "$path";; + esac + done + value=$(read_env "$WINDSHIFT_CONTAINER" PLUGIN_DIRS) || return 1 + while [ -n "$value" ]; do + case "$value" in *,*) path=${value%%,*}; value=${value#*,};; *) path=$value; value=;; esac + path=$(trim "$path") + [ -z "$path" ] || inside_data "$path" PLUGIN_DIRS + done +} +postgres_service() { + local candidate + if [ -n "$POSTGRES_SERVICE" ]; then valid_service_name "$POSTGRES_SERVICE" || return 1; require_container "$POSTGRES_SERVICE" >/dev/null; printf '%s\n' "$POSTGRES_SERVICE"; return; fi + for candidate in postgres db; do + if container "$candidate" >/dev/null 2>&1; then printf '%s\n' "$candidate"; return; fi + done + return 1 +} +valid_service_name() { + case "$1" in ''|-*|*[!A-Za-z0-9_.-]*) return 1;; esac +} +validate_postgres_service() { + local service=$1 host port app_user app_db db_container db_user db_db + host=$(read_env "$WINDSHIFT_CONTAINER" POSTGRES_HOST) || return 1 + port=$(read_env "$WINDSHIFT_CONTAINER" POSTGRES_PORT) || return 1 + app_user=$(read_env "$WINDSHIFT_CONTAINER" POSTGRES_USER) || return 1 + app_db=$(read_env "$WINDSHIFT_CONTAINER" POSTGRES_DB) || return 1 + [ "$host" = "$service" ] && { [ -z "$port" ] || [ "$port" = 5432 ]; } && [ -n "$app_user" ] && [ -n "$app_db" ] || return 1 + db_container=$(container "$service") || return 1 + db_user=$(read_env "$db_container" POSTGRES_USER) || return 1 + db_db=$(read_env "$db_container" POSTGRES_DB) || return 1 + [ "$app_user" = "$db_user" ] && [ "$app_db" = "$db_db" ] +} +acquire_lock() { + local labels project service key candidate + labels=$(docker inspect -f '{{index .Config.Labels "com.docker.compose.project"}} {{index .Config.Labels "com.docker.compose.service"}}' "$WINDSHIFT_CONTAINER") || die "could not read Compose labels for lock" + project=${labels%% *}; service=${labels#* } + [ -n "$project" ] && [ "$service" = windshift ] || die "container is not an identifiable Compose windshift service" + key=$(printf 'windshift-backup-lock-v1\0%s\0%s' "$project" "$service" | sha256_stdin) || die "could not hash lock identity" + candidate="/var/tmp/windshift-backup-$key.lock" + if ! mkdir -m 700 "$candidate" 2>/dev/null; then + [ -e "$candidate" ] || die "could not create system lock directory: $candidate" + die "backup/restore lock exists: $candidate. Confirm no operation is running, then remove only that stale lock directory" + fi + LOCK_DIR=$candidate + LOCK_PHYSICAL=$(physical_existing_dir "$LOCK_DIR") || die "could not resolve newly created recovery lock" + require_outside_data_mount "$LOCK_PHYSICAL" "backup/restore lock" + printf '%s\n' "$$" >"$LOCK_DIR/pid" +} +release_lock() { + if [ -n "$LOCK_DIR" ]; then + if ! rm -f "$LOCK_DIR/pid"; then warn "could not remove lock owner file $LOCK_DIR/pid" + elif ! rmdir "$LOCK_DIR"; then warn "could not remove nonempty lock $LOCK_DIR" + fi + fi + LOCK_DIR= LOCK_PHYSICAL= +} +stop_app() { + WAS_RUNNING=false + case "$(container_running_state "$WINDSHIFT_CONTAINER")" in + false) return 0;; + true) WAS_RUNNING=true; "${COMPOSE[@]}" stop --timeout 60 "$SERVICE" || return 1; stopped "$WINDSHIFT_CONTAINER";; + *) return 1;; + esac +} +stop_for_rollback() { + case "$(container_running_state "$WINDSHIFT_CONTAINER")" in + false) return 0;; + true) "${COMPOSE[@]}" stop --timeout 60 "$SERVICE" || return 1; stopped "$WINDSHIFT_CONTAINER";; + *) return 1;; + esac +} +start_and_verify() { + [ "$WAS_RUNNING" = true ] || return 0 + "${COMPOSE[@]}" start "$SERVICE" || return 1 + local status deadline=$((SECONDS + HEALTH_TIMEOUT)) + status=$(health_status "$WINDSHIFT_CONTAINER") || return 1 + if [ -z "$status" ]; then + while ((SECONDS < deadline)); do + running "$WINDSHIFT_CONTAINER" || return 1 + "${COMPOSE[@]}" exec -T "$SERVICE" /windshift healthcheck && return 0 + ((SECONDS < deadline)) || break + sleep 1 + done + return 1 + fi + while ((SECONDS < deadline)); do + running "$WINDSHIFT_CONTAINER" || return 1 + status=$(health_status "$WINDSHIFT_CONTAINER") || return 1 + [ "$status" = healthy ] && return 0 + [ "$status" = unhealthy ] && return 1 + ((SECONDS < deadline)) || break + sleep 1 + done + return 1 +} +archive_data() { + local dest=$1 output="$1/data.tar.gz" + (umask 077; : >"$output") || return 1 + chmod 600 "$output" || return 1 + docker run --rm --network none --read-only --security-opt no-new-privileges --volumes-from "$WINDSHIFT_CONTAINER:ro" "$HELPER_IMAGE" sh -ec 'tar -C "$1" -czf - .' sh "$DATA_PATH" >"$output" +} +verify_data_archive() { + local source=$1 sqlite_rel=${2:-} helper + read -r -d '' helper <<'EOF' || true +test -s /backup/data.tar.gz +tar -tzf /backup/data.tar.gz >/dev/null +tar -tvzf /backup/data.tar.gz >/dev/null +tar -tzf /backup/data.tar.gz | awk ' +{ + path = $0 + if (path == "./") key = "." + else { + if (substr(path, 1, 2) != "./") exit 1 + path = substr(path, 3) + if (path == "" || path ~ /(^|\/)\.($|\/)/ || path ~ /(^|\/)\.\.($|\/)/ || path ~ /\/\//) exit 1 + sub(/\/$/, "", path) + if (path == "") exit 1 + key = path + } + if (seen[key]++) exit 1 + count++ +} +END { if (!count) exit 1 } +' +if tar -tzf /backup/data.tar.gz | grep -Eq '(^/|(^|/)\.\.($|/))'; then exit 1; fi +if tar -tvzf /backup/data.tar.gz | grep -Eqv '^[-d]'; then exit 1; fi +if tar -tvzf /backup/data.tar.gz | grep -E '^-.* -> '; then exit 1; fi +tar -tzf /backup/data.tar.gz | grep -q . +EOF + if [ -n "$sqlite_rel" ]; then + helper+=$'\nsqlite_path=$1\ntar -tzf /backup/data.tar.gz | grep -Fx "./$sqlite_path" >/dev/null\ntar -tvzf /backup/data.tar.gz | awk -v path="./$sqlite_path" \'$NF == path && substr($1, 1, 1) == "-" { found = 1 } END { exit !found }\'\ntar -xOf /backup/data.tar.gz "./$sqlite_path" | wc -c | awk \'{ bytes += $1 } END { exit !(bytes > 0) }\'' + fi + docker run --rm --network none --read-only --security-opt no-new-privileges -v "$source:/backup:ro" "$HELPER_IMAGE" sh -ec "$helper" sh "$sqlite_rel" +} +extract_data() { + local source=$1 + docker run --rm --network none --read-only --security-opt no-new-privileges --volumes-from "$WINDSHIFT_CONTAINER" -v "$source:/backup:ro" "$HELPER_IMAGE" sh -ec 'mkdir -p "$1"; find "$1" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +; tar -C "$1" -xzf /backup/data.tar.gz' sh "$DATA_PATH" +} +dump_postgres() { + (umask 077; "${COMPOSE[@]}" exec -T "$2" sh -ec 'pg_dump --format=custom --no-owner --no-privileges -U "$POSTGRES_USER" "$POSTGRES_DB"' >"$1") && chmod 600 "$1" +} +restore_postgres() { + local dump=$1 service=$2 db_container token remote + [ -f "$dump" ] && [ ! -L "$dump" ] && [ -s "$dump" ] || return 2 + db_container=$(container "$service") || return 1 + token="$(date +%s)-$$-${RANDOM}" + remote="/tmp/windshift-restore-$token.dump" + PG_REMOTE=$remote + PG_REMOTE_SERVICE=$service + if ! docker cp "$dump" "$db_container:$remote"; then + if "${COMPOSE[@]}" exec -T "$service" sh -ec "rm -f '$remote'"; then PG_REMOTE=; PG_REMOTE_SERVICE=; else warn "could not remove a possibly partial PostgreSQL dump $remote"; fi + return 2 + fi + if ! "${COMPOSE[@]}" exec -T "$service" sh -ec "pg_restore --list '$remote' >/dev/null"; then + if "${COMPOSE[@]}" exec -T "$service" sh -ec "rm -f '$remote'"; then PG_REMOTE=; PG_REMOTE_SERVICE=; else warn "could not remove invalid PostgreSQL dump $remote"; fi + return 2 + fi + PHASE=pg-mutating + "${COMPOSE[@]}" exec -T "$service" sh -ec "pg_restore --clean --if-exists --no-owner --no-privileges --single-transaction --exit-on-error -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" '$remote'" || return 1 + if "${COMPOSE[@]}" exec -T "$service" sh -ec "rm -f '$remote'"; then PG_REMOTE=; PG_REMOTE_SERVICE=; else warn "could not remove temporary PostgreSQL dump $remote"; fi + return 0 +} +validate_postgres_dump() { + local dump=$1 service=$2 db_container token remote valid=true + [ -f "$dump" ] && [ ! -L "$dump" ] && [ -s "$dump" ] || return 1 + db_container=$(container "$service") || return 1 + token="$(date +%s)-$$-${RANDOM}" + remote="/tmp/windshift-validate-$token.dump" + PG_REMOTE=$remote + PG_REMOTE_SERVICE=$service + if ! docker cp "$dump" "$db_container:$remote"; then + if "${COMPOSE[@]}" exec -T "$service" sh -ec "rm -f '$remote'"; then PG_REMOTE=; PG_REMOTE_SERVICE=; else warn "could not remove a possibly partial PostgreSQL validation dump $remote"; fi + return 1 + fi + "${COMPOSE[@]}" exec -T "$service" sh -ec "pg_restore --list '$remote' >/dev/null" || valid=false + if "${COMPOSE[@]}" exec -T "$service" sh -ec "rm -f '$remote'"; then PG_REMOTE=; PG_REMOTE_SERVICE=; else warn "could not remove temporary PostgreSQL validation dump $remote"; return 1; fi + [ "$valid" = true ] +} +remove_unmutated_pg_remote() { + [ -n "$PG_REMOTE" ] || return 0 + [ -n "$PG_REMOTE_SERVICE" ] || return 1 + if "${COMPOSE[@]}" exec -T "$PG_REMOTE_SERVICE" sh -ec "rm -f '$PG_REMOTE'"; then PG_REMOTE=; PG_REMOTE_SERVICE=; return 0; fi + return 1 +} +manifest() { + local dest=$1 type=$2 service=$3 fingerprint=$4 source=$5 included=$6 + { + printf 'format_version=%s\ncreated_at=%s\ndatabase_type=%s\ndata_path=%s\npostgres_service=%s\nsecret_source=%s\nsecret_fingerprint=%s\nsso_secret_included=%s\nsqlite_db_path=%s\n' "$FORMAT_VERSION" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$type" "$DATA_PATH" "$service" "$source" "$fingerprint" "$included" "${SQLITE_DB_REL:-}" + } >"$dest/manifest.env" && chmod 600 "$dest/manifest.env" +} +write_sso_secret() { local secret; secret=$(effective_secret) || return 1; printf '%s' "$secret" >"$1/sso-secret" && chmod 600 "$1/sso-secret"; } +checksums() { + local dest=$1 type=$2 included=$3 f sum + : >"$dest/checksums.sha256" || return 1 + for f in manifest.env data.tar.gz; do sum=$(sha256 "$dest/$f") || return 1; printf '%s %s\n' "$sum" "$f" >>"$dest/checksums.sha256" || return 1; done + if [ "$type" = postgres ]; then sum=$(sha256 "$dest/database.dump") || return 1; printf '%s database.dump\n' "$sum" >>"$dest/checksums.sha256" || return 1; fi + if [ "$included" = true ]; then sum=$(sha256 "$dest/sso-secret") || return 1; printf '%s sso-secret\n' "$sum" >>"$dest/checksums.sha256" || return 1; fi + chmod 600 "$dest/checksums.sha256" +} +manifest_value() { awk -F= -v key="$2" '$1 == key {print substr($0,length(key)+2); exit}' "$1"; } +require_regular_artifacts() { + local source=$1 type=$2 included=$3 f + for f in manifest.env checksums.sha256 data.tar.gz; do [ -f "$source/$f" ] && [ ! -L "$source/$f" ] || return 1; done + [ "$type" != postgres ] || { [ -f "$source/database.dump" ] && [ ! -L "$source/database.dump" ]; } || return 1 + [ "$included" != true ] || { [ -f "$source/sso-secret" ] && [ ! -L "$source/sso-secret" ]; } || return 1 +} +verify() { + local source=$1 type included required expected file actual sqlite_rel + local seen_manifest=0 seen_data=0 seen_database=0 seen_secret=0 + [ -f "$source/manifest.env" ] && [ ! -L "$source/manifest.env" ] || return 1 + [ "$(manifest_value "$source/manifest.env" format_version)" = "$FORMAT_VERSION" ] || return 1 + type=$(manifest_value "$source/manifest.env" database_type); included=$(manifest_value "$source/manifest.env" sso_secret_included) + case "$type:$included" in sqlite:true|sqlite:false|postgres:true|postgres:false) ;; *) return 1;; esac + grep -Eq '^[[:xdigit:]]{64}$' < <(manifest_value "$source/manifest.env" secret_fingerprint) || return 1 + sqlite_rel=$(manifest_value "$source/manifest.env" sqlite_db_path) + [ "$type" != sqlite ] || { case "$sqlite_rel" in ''|*'..'*|*[!A-Za-z0-9._/-]*) return 1;; esac; } + require_regular_artifacts "$source" "$type" "$included" || return 1 + for required in manifest.env data.tar.gz; do grep -Eq "^[[:xdigit:]]{64} $required$" "$source/checksums.sha256" || return 1; done + [ "$type" != postgres ] || grep -Eq '^[[:xdigit:]]{64} database\.dump$' "$source/checksums.sha256" || return 1 + [ "$included" != true ] || grep -Eq '^[[:xdigit:]]{64} sso-secret$' "$source/checksums.sha256" || return 1 + while read -r expected file || [ -n "$expected$file" ]; do + case "$file" in + manifest.env) seen_manifest=$((seen_manifest + 1)); [ "$seen_manifest" = 1 ] || return 1;; + data.tar.gz) seen_data=$((seen_data + 1)); [ "$seen_data" = 1 ] || return 1;; + database.dump) seen_database=$((seen_database + 1)); [ "$seen_database" = 1 ] || return 1;; + sso-secret) seen_secret=$((seen_secret + 1)); [ "$seen_secret" = 1 ] || return 1;; + *) return 1;; + esac + actual=$(sha256 "$source/$file") || return 1 + [ "$actual" = "$expected" ] || return 1 + done <"$source/checksums.sha256" + [ "$seen_manifest" = 1 ] && [ "$seen_data" = 1 ] || return 1 + if [ "$type" = postgres ]; then [ "$seen_database" = 1 ] || return 1; else [ "$seen_database" = 0 ] || return 1; fi + if [ "$included" = true ]; then [ "$seen_secret" = 1 ] || return 1; else [ "$seen_secret" = 0 ] || return 1; fi + verify_data_archive "$source" "$sqlite_rel" || return 1 + printf '%s\n' "$type" +} +stage_backup() { + local source=$1 type included f + STAGE_DIR=$(mktemp -d "$ROLLBACK_ROOT/windshift-restore-stage.XXXXXX") || return 1 + chmod 700 "$STAGE_DIR" || return 1 + for f in manifest.env checksums.sha256; do [ -f "$source/$f" ] && [ ! -L "$source/$f" ] && cp -p "$source/$f" "$STAGE_DIR/$f" && [ -f "$STAGE_DIR/$f" ] && [ ! -L "$STAGE_DIR/$f" ] || return 1; done + type=$(manifest_value "$STAGE_DIR/manifest.env" database_type) + included=$(manifest_value "$STAGE_DIR/manifest.env" sso_secret_included) + case "$type:$included" in sqlite:true|sqlite:false|postgres:true|postgres:false) ;; *) return 1;; esac + require_regular_artifacts "$source" "$type" "$included" || return 1 + for f in data.tar.gz; do cp -p "$source/$f" "$STAGE_DIR/$f" && [ -f "$STAGE_DIR/$f" ] && [ ! -L "$STAGE_DIR/$f" ] || return 1; done + [ "$type" != postgres ] || { cp -p "$source/database.dump" "$STAGE_DIR/database.dump" && [ -f "$STAGE_DIR/database.dump" ] && [ ! -L "$STAGE_DIR/database.dump" ]; } || return 1 + [ "$included" != true ] || { cp -p "$source/sso-secret" "$STAGE_DIR/sso-secret" && [ -f "$STAGE_DIR/sso-secret" ] && [ ! -L "$STAGE_DIR/sso-secret" ]; } || return 1 + verify "$STAGE_DIR" >/dev/null +} +create_rollback() { + ROLLBACK_DIR=$(mktemp -d "$ROLLBACK_ROOT/windshift-restore-rollback.XXXXXX") || return 1 + chmod 700 "$ROLLBACK_DIR" || return 1 + archive_data "$ROLLBACK_DIR" || return 1 + verify_data_archive "$ROLLBACK_DIR" "${SQLITE_DB_REL:-}" || return 1 + ROLLBACK_CHECKSUM=$(sha256 "$ROLLBACK_DIR/data.tar.gz") || return 1 + : >"$ROLLBACK_DIR/checksums.sha256" || return 1 + printf '%s data.tar.gz\n' "$ROLLBACK_CHECKSUM" >>"$ROLLBACK_DIR/checksums.sha256" || return 1 + if [ "$LIVE_TYPE" = postgres ]; then + DB_ROLLBACK="$ROLLBACK_DIR/database-before.dump" + dump_postgres "$DB_ROLLBACK" "$PG_SERVICE" || return 1 + validate_postgres_dump "$DB_ROLLBACK" "$PG_SERVICE" || return 1 + DB_ROLLBACK_CHECKSUM=$(sha256 "$DB_ROLLBACK") || return 1 + printf '%s database-before.dump\n' "$DB_ROLLBACK_CHECKSUM" >>"$ROLLBACK_DIR/checksums.sha256" || return 1 + fi + chmod 600 "$ROLLBACK_DIR/checksums.sha256" +} +validate_rollback_data() { + [ -n "$ROLLBACK_DIR" ] && [ -f "$ROLLBACK_DIR/checksums.sha256" ] && [ ! -L "$ROLLBACK_DIR/checksums.sha256" ] && [ -f "$ROLLBACK_DIR/data.tar.gz" ] && [ ! -L "$ROLLBACK_DIR/data.tar.gz" ] && [ "$(sha256 "$ROLLBACK_DIR/data.tar.gz")" = "$ROLLBACK_CHECKSUM" ] && verify_data_archive "$ROLLBACK_DIR" "${SQLITE_DB_REL:-}" +} +restore_rollback_data() { + validate_rollback_data && extract_data "$ROLLBACK_DIR" +} +validate_rollback_postgres() { + [ -f "$DB_ROLLBACK" ] && [ ! -L "$DB_ROLLBACK" ] && [ -s "$DB_ROLLBACK" ] && [ "$(sha256 "$DB_ROLLBACK")" = "$DB_ROLLBACK_CHECKSUM" ] && validate_postgres_dump "$DB_ROLLBACK" "$PG_SERVICE" +} +restore_full_rollback() { + validate_rollback_data || return 1 + [ "$LIVE_TYPE" != postgres ] || validate_rollback_postgres || return 1 + extract_data "$ROLLBACK_DIR" || return 1 + [ "$LIVE_TYPE" != postgres ] || restore_postgres "$DB_ROLLBACK" "$PG_SERVICE" +} +cleanup_restore() { + local uncertainty + [ "$CLEANUP_RUNNING" = false ] || return + CLEANUP_RUNNING=true + trap - EXIT + trap '' INT TERM HUP + if [ "$ACTION" = backup ]; then + remove_unmutated_pg_remote || warn "could not remove unmutated PostgreSQL dump $PG_REMOTE" + if [ "$BACKUP_READY" = true ]; then + warn "valid backup retained after restart/health failure: $BACKUP_PATH" + elif [ "$BACKUP_DEST_CREATED" = true ]; then + rm -rf "$BACKUP_PATH" || warn "could not remove task-created partial backup $BACKUP_PATH" + fi + [ "$WAS_RUNNING" != true ] || start_and_verify || warn "backup failed and Windshift could not be restarted" + elif [ "$PHASE" = stopped-clean ] || [ "$PHASE" = preflight ]; then + remove_unmutated_pg_remote || warn "could not remove unmutated PostgreSQL dump $PG_REMOTE" + [ -z "$STAGE_DIR" ] || rm -rf "$STAGE_DIR" || warn "could not remove preflight stage $STAGE_DIR" + if [ -n "$ROLLBACK_DIR" ] && ! rm -rf "$ROLLBACK_DIR"; then warn "could not remove partial pre-mutation rollback $ROLLBACK_DIR"; fi + [ "$WAS_RUNNING" != true ] || start_and_verify || warn "operation stopped Windshift but could not restart it" + elif [ "$PHASE" = data-mutating ] && [ -n "$ROLLBACK_DIR" ]; then + remove_unmutated_pg_remote || warn "could not remove unmutated PostgreSQL dump $PG_REMOTE" + if ! restore_rollback_data; then + stop_for_rollback || warn "rollback failed and Windshift could not be proven stopped" + warn "restore aborted and /data rollback failed; Windshift remains stopped and the recovery lock is retained. rollback=$ROLLBACK_DIR staged=$STAGE_DIR lock=$LOCK_DIR" + return + fi + if start_and_verify; then + if [ "$WAS_RUNNING" = true ]; then + if ! rm -rf "$ROLLBACK_DIR" "$STAGE_DIR"; then warn "data rollback succeeded but retained artifacts at rollback=$ROLLBACK_DIR staged=$STAGE_DIR"; fi + warn "restore aborted during /data replacement; original /data was restored" + else + if ! rm -rf "$STAGE_DIR"; then warn "could not remove staged backup $STAGE_DIR"; fi + warn "restore aborted during /data replacement while Windshift was already stopped; no health check was possible. Retained rollback: $ROLLBACK_DIR" + fi + else + stop_for_rollback || warn "rolled-back Windshift could not be proven stopped after its health check failed" + warn "original /data was restored but failed its health check; Windshift remains stopped and the recovery lock is retained. rollback=$ROLLBACK_DIR staged=$STAGE_DIR lock=$LOCK_DIR" + return + fi + elif [ "$PHASE" = pg-mutating ] || [ "$PHASE" = post-restore ]; then + if [ "$LIVE_TYPE" = postgres ]; then uncertainty="PostgreSQL or /data state is uncertain"; else uncertainty="/data state is uncertain"; fi + if stop_for_rollback; then + warn "$uncertainty; Windshift is stopped and the recovery lock is retained. rollback=$ROLLBACK_DIR staged=$STAGE_DIR remote=$PG_REMOTE lock=$LOCK_DIR" + else + warn "$uncertainty and Windshift could not be proven stopped. Do not mutate rollback artifacts or remove the recovery lock. rollback=$ROLLBACK_DIR staged=$STAGE_DIR remote=$PG_REMOTE lock=$LOCK_DIR" + fi + return + fi + release_lock +} +on_exit() { local status=$?; cleanup_restore; exit "$status"; } +on_signal() { die "interrupted during $PHASE"; } +backup() { + local dest=$1 type service= fingerprint secret_source included=false sso physical_dest physical_rollback + WINDSHIFT_CONTAINER=$(require_container "$SERVICE") + data_mount_present || die "--data-path must be exactly one persistent mounted container destination" + physical_rollback=$(physical_existing_dir "$ROLLBACK_ROOT") || die "could not resolve rollback root" + physical_dest=$(physical_new_child_path "$dest") || die "backup directory parent must already exist and be resolvable" + require_outside_data_mount "$physical_rollback" "rollback root" + require_outside_data_mount "$physical_dest" "backup directory" + type=$(database_type) || die "deployment uses an unsupported database, SSH, entrypoint, or CLI override configuration" + check_paths "$type" || die "configured persistent paths are unsupported" + fingerprint=$(secret_fingerprint) || die "could not read and fingerprint SSO_SECRET/SESSION_SECRET" + sso=$(read_env "$WINDSHIFT_CONTAINER" SSO_SECRET) || die "could not read SSO_SECRET" + secret_source=SESSION_SECRET; [ -z "$sso" ] || secret_source=SSO_SECRET + if [ "$type" = postgres ]; then service=$(postgres_service) || die "no unambiguous PostgreSQL service"; validate_postgres_service "$service" || die "PostgreSQL settings do not prove a stock Compose service mapping"; fi + acquire_lock + case "$physical_dest" in "$LOCK_PHYSICAL"|"$LOCK_PHYSICAL"/*) die "backup directory must not resolve to the lock directory or a descendant: $dest";; esac + [ ! -e "$dest" ] || die "backup directory already exists: $dest" + mkdir -m 700 "$dest" || die "could not create backup directory" + BACKUP_DEST_CREATED=true + stop_app || die "could not stop Windshift within 60 seconds" + PHASE=backup-stopped + archive_data "$dest" || die "could not archive /data" + verify_data_archive "$dest" "${SQLITE_DB_REL:-}" || die "created data archive failed validation" + if [ "$type" = postgres ]; then + dump_postgres "$dest/database.dump" "$service" || die "could not dump PostgreSQL" + validate_postgres_dump "$dest/database.dump" "$service" || die "created PostgreSQL dump failed structural validation" + fi + [ "$INCLUDE_SSO_SECRET" != true ] || included=true + manifest "$dest" "$type" "$service" "$fingerprint" "$secret_source" "$included" || die "could not write manifest" + [ "$included" != true ] || write_sso_secret "$dest" || die "could not write sensitive secret file" + checksums "$dest" "$type" "$included" || die "could not create checksums" + verify "$dest" >/dev/null || die "created backup failed final verification" + PHASE=backup-ready + BACKUP_READY=true + start_and_verify || die "backup completed but Windshift did not become healthy" + release_lock + trap - EXIT INT TERM HUP + printf 'Backup created: %s\n' "$dest" +} +restore() { + local source=$1 type included expected actual manifest_service physical_source physical_rollback + [ -d "$source" ] || die "backup directory does not exist" + [ -f "$source/manifest.env" ] && [ ! -L "$source/manifest.env" ] || die "backup manifest is not a regular file" + WINDSHIFT_CONTAINER=$(require_container "$SERVICE") + data_mount_present || die "--data-path must be exactly one persistent mounted container destination" + physical_rollback=$(physical_existing_dir "$ROLLBACK_ROOT") || die "could not resolve rollback root" + physical_source=$(physical_existing_dir "$source") || die "could not resolve backup directory" + require_outside_data_mount "$physical_rollback" "rollback root" + require_outside_data_mount "$physical_source" "backup directory" + LIVE_TYPE=$(database_type) || die "deployment uses an unsupported database, SSH, entrypoint, or CLI override configuration" + check_paths "$LIVE_TYPE" || die "configured persistent paths are unsupported" + acquire_lock + stage_backup "$source" || die "could not create a verified private staged backup snapshot" + type=$(verify "$STAGE_DIR") || die "staged backup validation failed" + included=$(manifest_value "$STAGE_DIR/manifest.env" sso_secret_included) + [ "$LIVE_TYPE" = "$type" ] || die "staged backup database type does not match deployment" + [ "$type" != sqlite ] || [ "$(manifest_value "$STAGE_DIR/manifest.env" sqlite_db_path)" = "$SQLITE_DB_REL" ] || die "staged SQLite DB path does not match the live DB_PATH" + expected=$(manifest_value "$STAGE_DIR/manifest.env" secret_fingerprint); actual=$(secret_fingerprint) || die "could not fingerprint current secret" + [ "$actual" = "$expected" ] || die "backup uses a different SSO_SECRET/SESSION_SECRET; configure the source value first" + [ "$(manifest_value "$STAGE_DIR/manifest.env" data_path)" = "$DATA_PATH" ] || die "backup data path does not match" + if [ "$type" = postgres ]; then + manifest_service=$(manifest_value "$STAGE_DIR/manifest.env" postgres_service) + [ -n "$POSTGRES_SERVICE" ] || POSTGRES_SERVICE=$manifest_service + PG_SERVICE=$(postgres_service) || die "no unambiguous PostgreSQL service" + validate_postgres_service "$PG_SERVICE" || die "PostgreSQL settings do not prove a stock Compose service mapping" + validate_postgres_dump "$STAGE_DIR/database.dump" "$PG_SERVICE" || die "staged PostgreSQL dump failed structural preflight" + fi + stop_app || { PHASE=stopped-clean; die "could not stop Windshift within 60 seconds"; } + PHASE=stopped-clean + create_rollback || die "could not create complete verified rollback artifacts" + PHASE=data-mutating + extract_data "$STAGE_DIR" || die "target data extraction failed" + if [ "$type" = postgres ]; then + if restore_postgres "$STAGE_DIR/database.dump" "$PG_SERVICE"; then :; else + result=$? + [ "$result" = 2 ] && die "PostgreSQL dump copy or structural validation failed before database mutation" + PHASE=pg-mutating + die "PostgreSQL restore failed; state is uncertain" + fi + fi + PHASE=post-restore + if ! start_and_verify; then + stop_for_rollback || die "restored state was unhealthy and could not be stopped; rollback was not attempted. rollback=$ROLLBACK_DIR staged=$STAGE_DIR" + if restore_full_rollback && start_and_verify; then + PHASE=rolled-back + if ! rm -rf "$ROLLBACK_DIR" "$STAGE_DIR"; then warn "rollback succeeded but retained artifacts at rollback=$ROLLBACK_DIR staged=$STAGE_DIR"; fi + release_lock + trap - EXIT INT TERM HUP + die "restored state was unhealthy; previous state was restored" + fi + die "restored state was unhealthy and rollback could not be proven; Windshift remains stopped. rollback=$ROLLBACK_DIR staged=$STAGE_DIR" + fi + PHASE=complete + if [ "$WAS_RUNNING" != true ]; then + if ! rm -rf "$STAGE_DIR"; then warn "restore completed while stopped but retained staged backup at $STAGE_DIR"; fi + release_lock + trap - EXIT INT TERM HUP + printf 'Restore completed while Windshift was already stopped; no health check was possible. Retained rollback: %s\n' "$ROLLBACK_DIR" + return 0 + fi + if ! rm -rf "$ROLLBACK_DIR" "$STAGE_DIR"; then warn "restore is healthy but rollback/staging cleanup failed; retain private artifacts at rollback=$ROLLBACK_DIR staged=$STAGE_DIR"; fi + release_lock + trap - EXIT INT TERM HUP + printf 'Restore completed from: %s\n' "$source" +} + +while [ "$#" -gt 0 ]; do + case "$1" in + backup|restore) [ -z "$ACTION" ] || die "choose one action"; ACTION=$1;; + --compose-file|--data-path|--postgres-service|--helper-image|--rollback-root) [ "$#" -ge 2 ] || die "$1 needs a value"; case "$1" in --compose-file) COMPOSE+=(-f "$2");; --data-path) DATA_PATH=$2;; --postgres-service) POSTGRES_SERVICE=$2;; --helper-image) HELPER_IMAGE=$2;; --rollback-root) ROLLBACK_ROOT=$2;; esac; shift;; + --include-sso-secret) INCLUDE_SSO_SECRET=true;; + --force) FORCE=true;; + -h|--help) usage; exit 0;; + -*) die "unknown option: $1";; + *) [ -z "$BACKUP_PATH" ] || die "only one backup directory may be supplied"; BACKUP_PATH=$1;; + esac + shift +done +[ -n "$ACTION" ] && [ -n "$BACKUP_PATH" ] || { usage >&2; exit 1; } +DATA_PATH=$(normalise_container_path "$DATA_PATH") || die "unsafe --data-path" +BACKUP_PATH=$(normalise_host_path "$BACKUP_PATH") || die "backup path must be absolute, normalized, and contain no ':'" +ROLLBACK_ROOT=$(normalise_host_path "$ROLLBACK_ROOT") || die "rollback root must be absolute, normalized, and contain no ':'" +[ -d "$ROLLBACK_ROOT" ] && [ -w "$ROLLBACK_ROOT" ] || die "rollback root must exist and be writable: $ROLLBACK_ROOT" +case "$HELPER_IMAGE" in ''|-*) die "helper image must be a nonempty image reference and must not start with '-'";; esac +[ "$ACTION" != restore ] || [ "$FORCE" = true ] || die "restore requires --force" +[ "$ACTION" != restore ] || [ "$INCLUDE_SSO_SECRET" != true ] || die "--include-sso-secret is backup-only" +case "$HEALTH_TIMEOUT" in ''|*[!0-9]*) die "WINDSHIFT_BACKUP_HEALTH_TIMEOUT must be an integer from 3 to 3600";; esac +[ "$HEALTH_TIMEOUT" -ge 3 ] && [ "$HEALTH_TIMEOUT" -le 3600 ] || die "WINDSHIFT_BACKUP_HEALTH_TIMEOUT must be from 3 to 3600 seconds" +need docker; need awk; need cp; need mktemp; command -v sha256sum >/dev/null 2>&1 || command -v shasum >/dev/null 2>&1 || die "required command not found: sha256sum or shasum" +trap on_exit EXIT +trap on_signal INT TERM HUP +"$ACTION" "$BACKUP_PATH" diff --git a/deploy/docker-compose-main.yml b/deploy/docker-compose-main.yml index 7e8c40989..55b3680bd 100644 --- a/deploy/docker-compose-main.yml +++ b/deploy/docker-compose-main.yml @@ -25,6 +25,7 @@ services: - SSO_SECRET=${SSO_SECRET} - SESSION_IP_BINDING=${SESSION_IP_BINDING:-log} - ATTACHMENT_PATH=/data/attachments + - PLUGIN_DIR=/data/plugins - DB_TYPE=postgres - POSTGRES_HOST=db - POSTGRES_PORT=5432 diff --git a/deploy/test-backup.sh b/deploy/test-backup.sh new file mode 100755 index 000000000..ffeb1617e --- /dev/null +++ b/deploy/test-backup.sh @@ -0,0 +1,561 @@ +#!/usr/bin/env bash +# Focused Docker-free contract test for deploy/backup.sh. +set -euo pipefail +export WINDSHIFT_BACKUP_HEALTH_TIMEOUT=3 + +repo_root=$(cd "$(dirname "$0")/.." && pwd) +test_tmp=${TMPDIR:-/tmp} +test_root=$(mktemp -d "${test_tmp%/}/windshift-backup-test.XXXXXX") +lock_path= +cleanup() { + if [ -n "$lock_path" ]; then rm -f "$lock_path/pid" 2>/dev/null || true; rmdir "$lock_path" 2>/dev/null || true; fi + rm -rf "$test_root" +} +trap cleanup EXIT +state="$test_root/state" +bin="$test_root/bin" +mkdir -p "$state/windshift/data/attachments" "$bin" +printf 'original database\n' >"$state/windshift/data/windshift.db" +printf 'original upload\n' >"$state/windshift/data/attachments/file.txt" +printf 'backup-secret\n' >"$state/secret" +printf 'false\n' >"$state/stopped" + +cat >"$bin/docker" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +state=${FAKE_DOCKER_STATE:?} +printf '%s\n' "$*" >>"$state/docker-args" + +if [ "$1" = compose ]; then + shift + case "$1" in + ps) + case "${4:-}" in + windshift) printf 'windshift-id\n';; + customdb) [ -f "$state/db-type" ] && printf 'customdb-id\n';; + otherdb) [ -f "$state/db-type" ] && printf 'otherdb-id\n';; + esac + ;; + stop) printf 'true\n' >"$state/stopped"; rm -f "$state/active-health-failure";; + start) + printf 'false\n' >"$state/stopped" + [ ! -f "$state/arm-health-failure" ] || mv "$state/arm-health-failure" "$state/active-health-failure" + ;; + exec) + if [[ "$*" == *'/windshift healthcheck'* ]] && [ -f "$state/active-health-failure" ]; then exit 1; fi + if [[ "$*" == *'rm -f '* ]] && [[ "$*" == *'windshift-restore-'* ]] && [ -f "$state/fail-pg-cleanup" ]; then exit 1; fi + if [[ "$*" == *'pg_dump '* ]]; then + printf 'fake-postgresql-dump\n' + elif [[ "$*" == *'pg_restore --list '* ]]; then + grep -qx fake-postgresql-dump "$state/remote-dump" + elif [[ "$*" == *'pg_restore --clean '* ]]; then + [ ! -f "$state/fail-pg-restore" ] || exit 1 + grep -qx fake-postgresql-dump "$state/remote-dump" + elif [[ "$*" == *'rm -f '* ]]; then + rm -f "$state/remote-dump" + else + printf 'fake PostgreSQL command\n' + fi + ;; + *) exit 2;; + esac + exit +fi + +if [ "$1" = inspect ]; then + if [ "$3" = '{{.State.Running}}' ]; then + if [ "$(cat "$state/stopped")" = true ]; then printf 'false\n'; else printf 'true\n'; fi + elif [ "$3" = '{{if .State.Health}}{{.State.Health.Status}}{{end}}' ]; then + printf '\n' + elif [[ "$3" == *'.Config.Cmd'* ]]; then + [ ! -f "$state/cmd-args" ] || cat "$state/cmd-args" + elif [[ "$3" == *'.Config.Entrypoint'* ]]; then + if [ -f "$state/custom-entrypoint" ]; then printf '/custom-entrypoint\n'; else printf '/windshift\n'; fi + elif [[ "$3" == *'__WINDSHIFT_MOUNT_END__'* ]]; then + mount_source="$state/windshift/data" + [ ! -f "$state/mount-source" ] || mount_source=$(cat "$state/mount-source") + printf '/data\0%s\0' "$mount_source" + [ ! -f "$state/nested-mount" ] || printf '/data/attachments\0%s\0' "$state/nested-attachments" + printf '__WINDSHIFT_MOUNT_END__\0' + elif [[ "$3" == *'com.docker.compose.project'* ]]; then + printf '%s windshift\n' "${FAKE_COMPOSE_PROJECT:?}" + elif [ "$4" = customdb-id ] || [ "$4" = otherdb-id ]; then + if [[ "$3" == *'__WINDSHIFT_ENV_END__'* ]]; then + printf '%s\0' POSTGRES_USER=windshift POSTGRES_DB=windshift + printf '__WINDSHIFT_ENV_END__\0' + else + printf '%s\n' POSTGRES_USER=windshift POSTGRES_DB=windshift + fi + else + db_path=/data/windshift.db + plugin_dir=/data/plugins + [ ! -f "$state/empty-db-path" ] || db_path= + [ ! -f "$state/empty-plugin-dir" ] || plugin_dir= + if [[ "$3" == *'__WINDSHIFT_ENV_END__'* ]]; then + printf 'SSO_SECRET=%s\0' "$(cat "$state/secret")" + printf 'DB_PATH=%s\0' "$db_path" + printf '%s\0' ATTACHMENT_PATH=/data/attachments AI_PROMPTS_DIR=/data/prompts + printf 'PLUGIN_DIR=%s\0' "$plugin_dir" + [ ! -f "$state/db-type" ] || printf '%s\0' DB_TYPE=postgres "POSTGRES_HOST=$(cat "$state/pg-host")" POSTGRES_PORT=5432 POSTGRES_USER=windshift POSTGRES_DB=windshift + [ ! -f "$state/connection" ] || printf '%s\0' POSTGRES_CONNECTION_STRING=postgres://unsupported + printf '__WINDSHIFT_ENV_END__\0' + exit + fi + printf 'SSO_SECRET=%s\n' "$(cat "$state/secret")" + printf 'DB_PATH=%s\n' "$db_path" + printf '%s\n' ATTACHMENT_PATH=/data/attachments AI_PROMPTS_DIR=/data/prompts + printf 'PLUGIN_DIR=%s\n' "$plugin_dir" + [ ! -f "$state/db-type" ] || printf '%s\n' DB_TYPE=postgres POSTGRES_HOST="$(cat "$state/pg-host")" POSTGRES_PORT=5432 POSTGRES_USER=windshift POSTGRES_DB=windshift + [ ! -f "$state/connection" ] || printf '%s\n' POSTGRES_CONNECTION_STRING=postgres://unsupported + fi + exit +fi + +if [ "$1" = cp ]; then + if [ -f "$state/fail-pg-cp" ] && [[ "$3" == *'windshift-restore-'* ]]; then exit 1; fi + cp "$2" "$state/remote-dump" + exit 0 +fi + +if [ "$1" = run ]; then + backup= + script= + for arg in "$@"; do + case "$arg" in + *:/backup|*:/backup:ro) backup=${arg%%:/backup*};; + *'tar -C '*|*'tar -tzf '*) script=$arg;; + esac + done + [ -n "$backup" ] || [[ "$script" == *'-czf -'* ]] || exit 2 + if [[ "$script" == *'-czf -'* ]]; then + tar -C "$state/windshift/data" -czf - . + elif [[ "$script" == *'-czf /backup/data.tar.gz'* ]]; then + tar -C "$state/windshift/data" -czf "$backup/data.tar.gz" . + elif [[ "$script" == *'-xzf /backup/data.tar.gz'* ]]; then + if [ -f "$state/fail-extract-always" ]; then + find "$state/windshift/data" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} + + printf 'partial restore\n' >"$state/windshift/data/windshift.db" + exit 1 + fi + if [ -f "$state/fail-extract-once" ]; then + rm -f "$state/fail-extract-once" + find "$state/windshift/data" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} + + printf 'partial restore\n' >"$state/windshift/data/windshift.db" + exit 1 + fi + find "$state/windshift/data" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} + + tar -C "$state/windshift/data" -xzf "$backup/data.tar.gz" + else + translated=${script//\/backup/$backup} + sh -ec "$translated" sh "${!#}" + fi + exit +fi +exit 2 +EOF +chmod +x "$bin/docker" + +export FAKE_DOCKER_STATE="$state" +export FAKE_COMPOSE_PROJECT="windshift-test-${test_root##*.}" +export WINDSHIFT_BACKUP_ROLLBACK_ROOT="$test_root" +export PATH="$bin:$PATH" +hash_file() { + if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | awk '{print $1}'; else shasum -a 256 "$1" | awk '{print $1}'; fi +} +hash_stdin() { + if command -v sha256sum >/dev/null 2>&1; then sha256sum | awk '{print $1}'; else shasum -a 256 | awk '{print $1}'; fi +} +lock_key=$(printf 'windshift-backup-lock-v1\0%s\0windshift' "$FAKE_COMPOSE_PROJECT" | hash_stdin) +grep -Fq -- '- PLUGIN_DIR=/data/plugins' "$repo_root/deploy/docker-compose-main.yml" +existing="$test_root/existing" +mkdir "$existing" +printf 'keep\n' >"$existing/marker" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$existing"; then + printf 'backup over existing destination unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx keep "$existing/marker" +backup="$test_root/backup" +bash "$repo_root/deploy/backup.sh" backup --include-sso-secret --helper-image fake-helper "$backup" +grep -qx database_type=sqlite "$backup/manifest.env" +grep -qx sso_secret_included=true "$backup/manifest.env" +grep -qx backup-secret "$backup/sso-secret" +grep -Fq -- '--volumes-from windshift-id:ro' "$state/docker-args" +[ "$(cat "$state/stopped")" = false ] + +printf 'line one\nline two\n' >"$state/secret" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/multiline-secret"; then + printf 'backup with a multiline secret unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/multiline-secret" ] +[ "$(cat "$state/stopped")" = false ] +printf 'backup-secret\n' >"$state/secret" + +printf '%s\n' '--postgres-replica-count=2' >"$state/cmd-args" +bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/allowed-cli" +printf '%s\n' '--db=/outside/windshift.db' >"$state/cmd-args" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/blocked-cli"; then + printf 'backup with a CLI path override unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/blocked-cli" ] +rm "$state/cmd-args" + +touch "$state/custom-entrypoint" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/custom-entrypoint"; then + printf 'backup with a custom entrypoint unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/custom-entrypoint" ] +rm "$state/custom-entrypoint" + +if bash "$repo_root/deploy/backup.sh" backup --data-path /other --helper-image fake-helper "$test_root/unmounted-data"; then + printf 'backup with an unmounted data path unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/unmounted-data" ] + +touch "$state/nested-mount" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/nested-mount"; then + printf 'backup with a nested data mount unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/nested-mount" ] +rm "$state/nested-mount" + +printf '/var/tmp\n' >"$state/mount-source" +lock_path="/var/tmp/windshift-backup-$lock_key.lock" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/lock-in-data-mount"; then + printf 'backup with its lock inside the live data mount unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/lock-in-data-mount" ] +[ ! -e "$lock_path" ] +lock_path= +rm "$state/mount-source" + +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$state/windshift/data/nested-backup"; then + printf 'backup inside the live data mount unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$state/windshift/data/nested-backup" ] + +if bash "$repo_root/deploy/backup.sh" backup --rollback-root "$state/windshift/data" --helper-image fake-helper "$test_root/nested-rollback-root"; then + printf 'backup with a rollback root inside the live data mount unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/nested-rollback-root" ] + +touch "$state/empty-db-path" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/empty-db-path"; then + printf 'backup with an empty SQLite DB_PATH unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/empty-db-path" ] +rm "$state/empty-db-path" + +touch "$state/empty-plugin-dir" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/empty-plugin-dir"; then + printf 'backup with an empty PLUGIN_DIR unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/empty-plugin-dir" ] +rm "$state/empty-plugin-dir" + +if bash "$repo_root/deploy/backup.sh" backup --helper-image --privileged "$test_root/helper-option"; then + printf 'backup with a helper image that looks like a Docker option unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/helper-option" ] + +printf 'changed database\n' >"$state/windshift/data/windshift.db" + +lock_path="/var/tmp/windshift-backup-$lock_key.lock" +mkdir "$lock_path" +mkdir "$test_root/alternate-rollback-root" +if bash "$repo_root/deploy/backup.sh" backup --rollback-root "$test_root/alternate-rollback-root" --helper-image fake-helper "$test_root/locked"; then + printf 'backup with lock contention unexpectedly succeeded\n' >&2 + exit 1 +fi +rmdir "$lock_path" +lock_path= + +nested_lock="/var/tmp/windshift-backup-$lock_key.lock" +ln -s "$nested_lock" "$test_root/lock-alias" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/lock-alias/result"; then + printf 'backup through a symlink to its lock directory unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$nested_lock" ] +rm "$test_root/lock-alias" + +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$nested_lock/result"; then + printf 'backup below its own lock directory unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$nested_lock" ] + +if bash "$repo_root/deploy/backup.sh" restore --helper-image fake-helper "$backup"; then + printf 'restore without --force unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx 'changed database' "$state/windshift/data/windshift.db" + +printf 'different-secret\n' >"$state/secret" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup"; then + printf 'restore with a different secret unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx 'changed database' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = false ] +printf 'backup-secret\n' >"$state/secret" + +unterminated_checksums="$test_root/unterminated-checksums" +cp -R "$backup" "$unterminated_checksums" +awk '$2 != "sso-secret" {print}' "$unterminated_checksums/checksums.sha256" >"$unterminated_checksums/checksums.next" +printf '%064d sso-secret' 0 >>"$unterminated_checksums/checksums.next" +mv "$unterminated_checksums/checksums.next" "$unterminated_checksums/checksums.sha256" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$unterminated_checksums"; then + printf 'restore with an unchecked unterminated checksum unexpectedly succeeded\n' >&2 + exit 1 +fi +[ "$(cat "$state/stopped")" = false ] + +duplicate_checksums="$test_root/duplicate-checksums" +cp -R "$backup" "$duplicate_checksums" +manifest_checksum=$(awk '$2 == "manifest.env" {print; exit}' "$duplicate_checksums/checksums.sha256") +printf '%s\n' "$manifest_checksum" >>"$duplicate_checksums/checksums.sha256" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$duplicate_checksums"; then + printf 'restore with a duplicate checksum entry unexpectedly succeeded\n' >&2 + exit 1 +fi +[ "$(cat "$state/stopped")" = false ] + +touch "$state/fail-extract-once" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup"; then + printf 'restore with extraction failure unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx 'changed database' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = false ] + +printf 'true\n' >"$state/stopped" +touch "$state/fail-extract-once" +stopped_failure_error="$test_root/stopped-data-failure.err" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup" 2>"$stopped_failure_error"; then + printf 'stopped restore with an extraction failure unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -Fq 'no health check was possible' "$stopped_failure_error" +stopped_failure_rollback=$(sed -n 's/.*Retained rollback: //p' "$stopped_failure_error" | tail -1) +[ -d "$stopped_failure_rollback" ] +grep -qx 'changed database' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = true ] +[ ! -e "$nested_lock" ] +if find "$test_root" -maxdepth 1 -type d -name 'windshift-restore-stage.*' -print -quit | grep -q .; then + printf 'staging directory was retained after a stopped rollback\n' >&2 + exit 1 +fi +rm -rf "$stopped_failure_rollback" +printf 'false\n' >"$state/stopped" + +touch "$state/fail-extract-always" +data_failure_error="$test_root/data-failure.err" +lock_path="$nested_lock" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup" 2>"$data_failure_error"; then + printf 'restore with a failed data rollback unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -Fq '/data rollback failed' "$data_failure_error" +grep -Fq 'recovery lock is retained' "$data_failure_error" +[ -d "$lock_path" ] +[ -f "$lock_path/pid" ] +[ "$(cat "$state/stopped")" = true ] +retained_rollback=$(sed -n 's/.*rollback=\([^ ]*\).*/\1/p' "$data_failure_error" | tail -1) +retained_stage=$(sed -n 's/.*staged=\([^ ]*\).*/\1/p' "$data_failure_error" | tail -1) +[ -d "$retained_rollback" ] +[ -d "$retained_stage" ] +rm -rf "$retained_rollback" "$retained_stage" +rm -f "$state/fail-extract-always" "$lock_path/pid" +rmdir "$lock_path" +lock_path= +printf 'changed database\n' >"$state/windshift/data/windshift.db" +printf 'false\n' >"$state/stopped" + +bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup" +grep -qx 'original database' "$state/windshift/data/windshift.db" +grep -qx 'original upload' "$state/windshift/data/attachments/file.txt" +[ "$(cat "$state/stopped")" = false ] + +printf 'state before failed readiness\n' >"$state/windshift/data/windshift.db" +touch "$state/arm-health-failure" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup"; then + printf 'restore with a failing readiness probe unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx 'state before failed readiness' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = false ] + +printf 'changed while stopped\n' >"$state/windshift/data/windshift.db" +printf 'true\n' >"$state/stopped" +stopped_output=$(bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup") +rollback_path=${stopped_output##*Retained rollback: } +[ -d "$rollback_path" ] +grep -qx 'original database' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = true ] +if find "$test_root" -maxdepth 1 -type d -name 'windshift-restore-stage.*' -print -quit | grep -q .; then + printf 'staging directory was retained after a stopped-service restore\n' >&2 + exit 1 +fi +rm -rf "$rollback_path" +printf 'false\n' >"$state/stopped" + +rewrite_data_checksum() { + local dir=$1 sum + sum=$(hash_file "$dir/data.tar.gz") + awk -v sum="$sum" '$2 == "data.tar.gz" {print sum " " $2; next} {print}' "$dir/checksums.sha256" >"$dir/checksums.next" + mv "$dir/checksums.next" "$dir/checksums.sha256" +} +mismatch="$test_root/sqlite-path-mismatch" +cp -R "$backup" "$mismatch" +mkdir "$mismatch/content" +tar -C "$mismatch/content" -xzf "$mismatch/data.tar.gz" +cp "$mismatch/content/windshift.db" "$mismatch/content/other.db" +tar -C "$mismatch/content" -czf "$mismatch/data.tar.gz" . +rm -rf "$mismatch/content" +rewrite_data_checksum "$mismatch" +awk -F= '$1 == "sqlite_db_path" {$0 = "sqlite_db_path=other.db"} {print}' "$mismatch/manifest.env" >"$mismatch/manifest.next" +mv "$mismatch/manifest.next" "$mismatch/manifest.env" +manifest_sum=$(hash_file "$mismatch/manifest.env") +awk -v sum="$manifest_sum" '$2 == "manifest.env" {print sum " " $2; next} {print}' "$mismatch/checksums.sha256" >"$mismatch/checksums.next" +mv "$mismatch/checksums.next" "$mismatch/checksums.sha256" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$mismatch"; then + printf 'restore with mismatched SQLite path unexpectedly succeeded\n' >&2 + exit 1 +fi +[ "$(cat "$state/stopped")" = false ] + +for kind in empty-db symlink hardlink duplicate; do + bad="$test_root/$kind" + cp -R "$backup" "$bad" + mkdir "$bad/content" + case "$kind" in + empty-db) : >"$bad/content/windshift.db";; + symlink) printf 'db\n' >"$bad/content/windshift.db"; ln -s windshift.db "$bad/content/link";; + hardlink) printf 'db\n' >"$bad/content/windshift.db"; ln "$bad/content/windshift.db" "$bad/content/alias";; + duplicate) printf 'db\n' >"$bad/content/windshift.db";; + esac + if [ "$kind" = duplicate ]; then tar -C "$bad/content" -czf "$bad/data.tar.gz" ./windshift.db ./windshift.db + else tar -C "$bad/content" -czf "$bad/data.tar.gz" . + fi + rewrite_data_checksum "$bad" + if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$bad"; then + printf 'restore of %s archive unexpectedly succeeded\n' "$kind" >&2 + exit 1 + fi + [ "$(cat "$state/stopped")" = false ] +done + +touch "$state/connection" +if bash "$repo_root/deploy/backup.sh" backup --helper-image fake-helper "$test_root/unsupported-connection"; then + printf 'backup with POSTGRES_CONNECTION_STRING unexpectedly succeeded\n' >&2 + exit 1 +fi +[ ! -e "$test_root/unsupported-connection" ] +rm "$state/connection" + +printf 'postgres\n' >"$state/db-type" +printf 'customdb\n' >"$state/pg-host" +postgres_backup="$test_root/postgres-backup" +bash "$repo_root/deploy/backup.sh" backup --postgres-service customdb --helper-image fake-helper "$postgres_backup" +grep -qx postgres_service=customdb "$postgres_backup/manifest.env" +grep -Fq 'pg_restore --list ' "$state/docker-args" +invalid_postgres="$test_root/invalid-postgres" +cp -R "$postgres_backup" "$invalid_postgres" +printf 'invalid dump\n' >"$invalid_postgres/database.dump" +database_sum=$(hash_file "$invalid_postgres/database.dump") +awk -v sum="$database_sum" '$2 == "database.dump" {print sum " " $2; next} {print}' "$invalid_postgres/checksums.sha256" >"$invalid_postgres/checksums.next" +mv "$invalid_postgres/checksums.next" "$invalid_postgres/checksums.sha256" +invalid_extracts_before=$(grep -c -- '-xzf /backup/data.tar.gz' "$state/docker-args" || true) +if bash "$repo_root/deploy/backup.sh" restore --force --postgres-service customdb --helper-image fake-helper "$invalid_postgres"; then + printf 'restore with a structurally invalid PostgreSQL dump unexpectedly succeeded\n' >&2 + exit 1 +fi +[ "$(cat "$state/stopped")" = false ] +invalid_extracts_after=$(grep -c -- '-xzf /backup/data.tar.gz' "$state/docker-args" || true) +[ "$invalid_extracts_after" -eq "$invalid_extracts_before" ] +printf 'otherdb\n' >"$state/pg-host" +bash "$repo_root/deploy/backup.sh" restore --force --postgres-service otherdb --helper-image fake-helper "$postgres_backup" +grep -Fq 'exec -T otherdb sh -ec pg_restore' "$state/docker-args" +touch "$state/fail-pg-cleanup" +bash "$repo_root/deploy/backup.sh" restore --force --postgres-service otherdb --helper-image fake-helper "$postgres_backup" +[ "$(cat "$state/stopped")" = false ] +rm "$state/fail-pg-cleanup" + +printf 'state before failed PostgreSQL readiness\n' >"$state/windshift/data/windshift.db" +pg_restores_before=$(grep -c 'pg_restore --clean ' "$state/docker-args" || true) +touch "$state/arm-health-failure" +if bash "$repo_root/deploy/backup.sh" restore --force --postgres-service otherdb --helper-image fake-helper "$postgres_backup"; then + printf 'PostgreSQL restore with a failing readiness probe unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx 'state before failed PostgreSQL readiness' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = false ] +pg_restores_after=$(grep -c 'pg_restore --clean ' "$state/docker-args" || true) +[ "$pg_restores_after" -eq "$((pg_restores_before + 2))" ] + +printf 'true\n' >"$state/stopped" +postgres_stopped_output=$(bash "$repo_root/deploy/backup.sh" restore --force --postgres-service otherdb --helper-image fake-helper "$postgres_backup") +postgres_rollback=${postgres_stopped_output##*Retained rollback: } +[ -d "$postgres_rollback" ] +grep -Eq '^[[:xdigit:]]{64} data\.tar\.gz$' "$postgres_rollback/checksums.sha256" +grep -Eq '^[[:xdigit:]]{64} database-before\.dump$' "$postgres_rollback/checksums.sha256" +[ "$(hash_file "$postgres_rollback/data.tar.gz")" = "$(awk '$2 == "data.tar.gz" {print $1}' "$postgres_rollback/checksums.sha256")" ] +[ "$(hash_file "$postgres_rollback/database-before.dump")" = "$(awk '$2 == "database-before.dump" {print $1}' "$postgres_rollback/checksums.sha256")" ] +rm -rf "$postgres_rollback" +printf 'false\n' >"$state/stopped" +rm "$state/db-type" "$state/pg-host" + +printf tampered >>"$backup/data.tar.gz" +if bash "$repo_root/deploy/backup.sh" restore --force --helper-image fake-helper "$backup"; then + printf 'restore of a tampered backup unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx 'original database' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = false ] + +printf 'postgres\n' >"$state/db-type" +printf 'customdb\n' >"$state/pg-host" +touch "$state/fail-pg-cp" +printf 'state before PostgreSQL copy failure\n' >"$state/windshift/data/windshift.db" +if bash "$repo_root/deploy/backup.sh" restore --force --postgres-service customdb --helper-image fake-helper "$postgres_backup"; then + printf 'restore with PostgreSQL copy failure unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -qx 'state before PostgreSQL copy failure' "$state/windshift/data/windshift.db" +[ "$(cat "$state/stopped")" = false ] +rm "$state/fail-pg-cp" + +touch "$state/fail-pg-restore" +pg_failure_error="$test_root/pg-failure.err" +lock_path="$nested_lock" +if bash "$repo_root/deploy/backup.sh" restore --force --postgres-service customdb --helper-image fake-helper "$postgres_backup" 2>"$pg_failure_error"; then + printf 'restore with a mutating PostgreSQL failure unexpectedly succeeded\n' >&2 + exit 1 +fi +grep -Fq 'recovery lock is retained' "$pg_failure_error" +[ -d "$lock_path" ] +[ -f "$lock_path/pid" ] +[ "$(cat "$state/stopped")" = true ] +retained_rollback=$(sed -n 's/.*rollback=\([^ ]*\).*/\1/p' "$pg_failure_error" | tail -1) +retained_stage=$(sed -n 's/.*staged=\([^ ]*\).*/\1/p' "$pg_failure_error" | tail -1) +[ -d "$retained_rollback" ] +[ -d "$retained_stage" ] +rm -rf "$retained_rollback" "$retained_stage" +rm -f "$state/remote-dump" "$state/fail-pg-restore" "$lock_path/pid" +rmdir "$lock_path" +lock_path= +printf 'false\n' >"$state/stopped" +rm "$state/db-type" "$state/pg-host" +printf 'backup script focused tests passed\n' From 70454c82fa97351f4145a5c7afb9c34bba64229e Mon Sep 17 00:00:00 2001 From: Optic00 <20187253+Optic00@users.noreply.github.com> Date: Sat, 29 Aug 2026 14:38:32 +0200 Subject: [PATCH 2/2] docs: clarify PostgreSQL rollback command --- deploy/backup-restore.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deploy/backup-restore.md b/deploy/backup-restore.md index c3bd1ee38..db8a5d84c 100644 --- a/deploy/backup-restore.md +++ b/deploy/backup-restore.md @@ -163,7 +163,7 @@ Set `POSTGRES_SERVICE` to the confirmed service name and keep `remote_dump` for [ "$(printf '%s\n' "$db_id" | sed '/^$/d' | wc -l | tr -d ' ')" = 1 ] || exit 1 remote_dump=/tmp/windshift-manual-rollback-$$.dump docker cp "$ROLLBACK_DIR/database-before.dump" "$db_id:$remote_dump" - docker compose exec -T "$POSTGRES_SERVICE" sh -ec 'pg_restore --list "$1" >/dev/null' sh "$remote_dump" + docker compose exec -T -e REMOTE_DUMP="$remote_dump" "$POSTGRES_SERVICE" sh -ec 'pg_restore --list "$REMOTE_DUMP" >/dev/null' For `/data`, resolve exactly one Windshift container before stopping it: @@ -180,7 +180,7 @@ For `/data`, resolve exactly one Windshift container before stopping it: For PostgreSQL, restore the already validated remote dump only after confirming the target database again: - docker compose exec -T "$POSTGRES_SERVICE" sh -ec 'pg_restore --clean --if-exists --no-owner --no-privileges --single-transaction --exit-on-error -U "$POSTGRES_USER" -d "$POSTGRES_DB" "$1"' sh "$remote_dump" + docker compose exec -T -e REMOTE_DUMP="$remote_dump" "$POSTGRES_SERVICE" sh -ec 'pg_restore --clean --if-exists --no-owner --no-privileges --single-transaction --exit-on-error -U "$POSTGRES_USER" -d "$POSTGRES_DB" "$REMOTE_DUMP"' docker compose exec -T "$POSTGRES_SERVICE" rm -f "$remote_dump" The PostgreSQL rollback cleans objects listed in its dump but may leave unrelated objects created by a newer schema.