Skip to content

Add Google Cloud Storage backup destination #29

Description

@immanuel-peter

Why This Matters

RedisBackup and RedisScheduledBackup already support S3-compatible storage and Azure Blob Storage. GKE/GCP users should be able to store Redis backup artifacts in Google Cloud Storage without routing through an S3 compatibility layer or maintaining separate storage infrastructure.

This should be implemented as a first-class GCS destination so upload, restore, validation, docs, and status paths all behave consistently with the existing S3 and Azure adapters.

Current State

  • BackupDestination currently supports exactly one of s3 or azure in api/v1/redisbackup_types.go.
  • The instance manager uploads backup artifacts from internal/instance-manager/webserver/backup.go via backend-specific branches.
  • Bootstrap restore runs through the separate init-container path in internal/cmd/manager/restore/restore.go, so restore needs its own GCS downloader branch.
  • Backup artifact status paths currently use s3://... and azblob://...; GCS should use gs://{bucket}/{object}.

Proposed API

Add a third destination:

spec:
  destination:
    gcs:
      bucket: my-redis-backups
      path: prod/redis # optional object prefix

Suggested Go shape:

type BackupDestination struct {
    S3    *S3Destination        `json:"s3,omitempty"`
    Azure *AzureBlobDestination `json:"azure,omitempty"`
    GCS   *GCSDestination       `json:"gcs,omitempty"`
}

type GCSDestination struct {
    Bucket string `json:"bucket"`
    Path   string `json:"path,omitempty"`
}

Update BackupDestination.Validate() so exactly one of s3, azure, or gcs is set, and require destination.gcs.bucket when GCS is selected.

Credential Model

For an initial parity implementation, keep the existing RedisCluster.spec.backupCredentialsSecret model and read credentials from the projected secret in the pod/init-container.

Support a service account JSON key from one or more conventional secret keys, for example:

  • GOOGLE_APPLICATION_CREDENTIALS_JSON
  • GOOGLE_SERVICE_ACCOUNT_JSON
  • serviceAccountJSON

Use the Google Cloud Storage Go client with explicit credentials, e.g. cloud.google.com/go/storage plus google.golang.org/api/option.WithCredentialsJSON.

Secretless GKE Workload Identity / ADC support is useful, but it changes the current restore contract because restore.Run() currently requires spec.backupCredentialsSecret before downloading. Treat Workload Identity support as a deliberate follow-up unless this issue explicitly expands scope.

Implementation Scope

  • Add GCSDestination to api/v1/redisbackup_types.go.
  • Update BackupDestination.Validate() and related tests in api/v1/redisbackup_types_test.go.
  • Run make generate and make manifests so zz_generated.deepcopy.go and CRDs include spec.destination.gcs.
  • Add a GCS upload branch in internal/instance-manager/webserver/backup.go.
  • Return backup metadata with backupPath: gs://{bucket}/{object} and the existing checksum/size fields.
  • Add a GCS restore branch in internal/cmd/manager/restore/restore.go.
  • Add a resolveGCSLocation(configuredBucket, backupPath) helper parallel to the S3/Azure location helpers.
  • Add downloadGCSObjectToFile(...) with the same temp-file-and-rename safety pattern as the existing download helpers.
  • Update backup docs in internal/controller/backup/README.md with the GCS spec, secret keys, status path format, and lifecycle-policy note.
  • Update retention comments/docs that currently mention only S3/Azure lifecycle policies.

Acceptance Criteria

  • A RedisBackup with spec.destination.gcs.bucket validates successfully.
  • A RedisBackup with missing destination.gcs.bucket is rejected by shared destination validation.
  • A backup request to the instance manager uploads the artifact to GCS using the configured path prefix.
  • Completed GCS backups record status.backupPath as gs://{bucket}/{object}.
  • Bootstrap restore can download both RDB and AOF archive artifacts from a completed GCS-backed RedisBackup.
  • Restore rejects mismatched gs:// bucket paths when the configured destination bucket differs from status.backupPath.
  • Generated CRDs expose spec.destination.gcs for both RedisBackup and RedisScheduledBackup.
  • Unit tests cover destination validation, GCS credential reading, gs:// path resolution, and upload/restore dispatch behavior.
  • make test and make lint pass locally.

Notes

Keep operator/controller credentials out of scope. The controller should continue to delegate artifact writes to the in-pod instance manager and should not gain object-store delete permissions. Remote artifact retention should remain an object-store lifecycle policy concern, now including GCS lifecycle rules for the configured prefix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions