Skip to content

internal/dave: improve snapshot archive storage logic #7

Description

@joshuasing

Currently, Dave uses a Repository to store snapshots. Internally, a Repository looks like this:

// Repository is a storage implementation which stores snapshots.
type Repository interface {
	Metadata(ctx context.Context) (*RepositoryMeta, error)
	MetadataUpdate(ctx context.Context, meta *RepositoryMeta) error

	SnapshotAdd(ctx context.Context, snapshot *Snapshot) error
	SnapshotByID(ctx context.Context, id string) (*Snapshot, error)
	SnapshotList(ctx context.Context) ([]*Snapshot, error)
	SnapshotRemove(ctx context.Context, id string) error
}

A Snapshot consists of metadata and one or more archives. This means that we only add a snapshot once all archives are created.

Archives are stored on the local file system before being moved to the repository, requiring enough storage for:

  1. The data being backed up (e.g. 1TB)
  2. The clone of data being backed up (same as 1; 1TB)
  3. All archives of the cloned data (same as 2, compressed, e.g. 0.8x)

In cases where we are making a snapshot of multiple very large directories, it may work better to upload snapshot archives individually to the repository as they are created, slightly reducing the storage required to use Dave.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions