dezhan

An air-gapped, immutable backup vault that speaks S3, with its integrity rules formally proved rather than merely tested.

Point any S3 client (aws-cli, restic, Veeam, Velero, boto3) at dezhan and your data is content-addressed, encrypted, erasure-coded, and WORM-locked: once an object is written under a retention it cannot be changed or deleted until that retention expires, and a tampered system clock cannot expire the lock early. It runs on-prem and offline, an alternative to MinIO and Veeam.

What sets it apart is the last part. The retention state machine, clock guard, audit chain, and erasure coding are written in SPARK and checked by gnatprove on every commit (325 checks, 0 unproved). "A retained object cannot be deleted before expiry" is a machine-proved theorem here, not a test that might have gaps.

Why dezhan

• Immutable by proof. WORM / Object Lock is enforced by a formally verified state machine: retention can be extended, never shortened.
• Speaks S3. A drop-in endpoint for the backup tools you already run, so there is no new client to learn.
• Air-gapped. Runs fully offline with no external runtime dependencies.
• Kubernetes-native. An operator runs vaults from a custom resource, and a CSI driver can expose a vault as PersistentVolumes.

Three images are published to GHCR: dezhan (the vault server), dezhan-operator (runs vaults from a custom resource), and dezhan-csi (exposes a vault as PersistentVolumes). For a plain install you only need the server, below.

Install

On-prem (pulls the image, runs it, prints generated credentials):

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/obsernetics/dezhan/main/install.sh | sh

Kubernetes operator (then create a DezhanVault, below):

kubectl apply -f https://raw.githubusercontent.com/obsernetics/dezhan/main/deploy/dezhan.yaml

Or with Helm from the chart repo (set vault.create=true to provision a vault too):

helm repo add dezhan https://obsernetics.github.io/dezhan
helm repo update
helm install dezhan dezhan/dezhan

Use it

ALIAS="aws --endpoint-url http://localhost:8080 --region us-east-1"
$ALIAS s3 mb s3://backups
$ALIAS s3 cp ./data.tar s3://backups/      # any size; multipart handled
$ALIAS s3 ls s3://backups/

# make a bucket immutable (WORM / Object Lock)
$ALIAS s3api create-bucket --bucket vault --object-lock-enabled-for-bucket
$ALIAS s3 cp important.bak s3://vault/
$ALIAS s3 rm  s3://vault/important.bak     # denied until retention expires

Standard S3 validated against the AWS SDK: buckets, objects, range reads, copy, batch delete, multipart, versioning with delete markers, user metadata, conditional requests, presigned URLs, SigV4, and Object Lock / WORM with legal hold. Retention is enforced by a formally verified state machine, and a tampered system clock cannot expire a lock.

Buckets are mutable (Standard) by default (overwrite in place, free delete); enabling Object Lock makes a bucket Immutable/WORM. For Kubernetes, example CronJobs orchestrate an air-gap sync window (deploy/airgap-sync.yaml) and back up etcd into a vault (deploy/etcd-backup.yaml).

CLI

A built-in client ships in the image and the binaries (DEZHAN_ADDR=host:port, default 127.0.0.1:8080):

dezhan_cli health                                  # ok / sealed
dezhan_cli put report "data" compliance 86400      # store under 1-day retention
dezhan_cli get report
dezhan_cli del report                              # denied until retention expires

More examples (aws-cli, restic, boto3, Velero, the operator CR) are in examples/.

Configuration

Variable	Meaning	Default
DEZHAN_VAULT_KEY	passphrase the data key is wrapped under	demo key
DEZHAN_REQUIRE_AUTH	reject unsigned requests	unset (anonymous)
DEZHAN_ACCESS_KEY / DEZHAN_SECRET	the S3 credential	dezhanadmin / demo
DEZHAN_CREDENTIALS	extra accesskey secret [default] [bucket:perm ...] lines	<root>/credentials
DEZHAN_TOKENS	API token / service-account lines token accesskey	<root>/tokens
DEZHAN_ADMIN_TOKEN	token (X-Dezhan-Admin-Token) gating /admin/*	unset
DEZHAN_DELETE_QUORUM / DEZHAN_APPROVERS	four-eyes deletes	0 / unset
DEZHAN_APPROVAL_TTL	seconds a staged delete approval stays valid	3600
DEZHAN_SCRUB_INTERVAL	seconds between integrity scrubs	300
DEZHAN_CHECKPOINT_INTERVAL	seconds between audit checkpoints (0 = off)	0
DEZHAN_GC_INTERVAL	seconds between garbage-collection passes (0 = off)	0

dezhan_server [port] [data-dir]. Operations: GET /healthz, GET /metrics (Prometheus), POST /admin/{seal,scrub,checkpoint}, web UI at /. Run sh scripts/smoke.sh for a boto3 conformance check.

Each credential has a default access level (rw, ro, or none) plus optional per-bucket overrides, e.g. auditor s3cret none logs:ro (no access except read-only on logs) or app s3cret ro data:rw (read everywhere, write to data). A write needs rw on that bucket; a read needs ro or rw.

Service accounts use API tokens: a DEZHAN_TOKENS line tok-abc app lets a client authenticate with Authorization: Bearer tok-abc, inheriting the app principal's policy (no SigV4 signing required).

With DEZHAN_DELETE_QUORUM set, a delete needs approver co-signatures, either synchronously (secrets in the X-Dezhan-Approvals header) or staged: approvers pre-authorize a specific object from separate sessions with POST /approve?resource=/bucket/key (header X-Dezhan-Approval: <secret>), and the delete succeeds once enough approvals accrue. Staged approvals expire after DEZHAN_APPROVAL_TTL and are consumed on use.

Kubernetes operator

Declare a DezhanVault; the operator reconciles it into a single-replica StatefulSet, a Service, a PVC, and a PodDisruptionBudget, and reports readiness on the resource status.

apiVersion: dezhan.obsernetics.io/v1alpha1
kind: DezhanVault
metadata:
  name: my-vault
spec:
  storage: 100Gi
  requireAuth: true
  deleteQuorum: 2                # deletes need 2 approver co-signatures
  secretName: my-vault-secrets   # DEZHAN_VAULT_KEY, DEZHAN_SECRET, ...

kubectl apply -f my-vault.yaml
kubectl get dezhanvaults          # READY, ENDPOINT, AGE

Reach the vault in-cluster at http://my-vault.<namespace>.svc:8080. A ready-to-edit sample is in operator/config/samples.

Spec

Field	Default	Meaning
image	ghcr.io/obsernetics/dezhan:latest	server image
port	8080	listen port
storage	50Gi	persistent volume size (raise it to expand online)
storageClassName	cluster default	PVC storage class
requireAuth	true	reject unsigned requests
deleteQuorum	0	approver co-signatures required to delete
scrubIntervalSeconds	0 (server default 300)	recurring verify-and-self-heal interval
secretName	none	Secret whose keys become server env vars
serviceType	ClusterIP	ClusterIP (headless) / NodePort / LoadBalancer
priorityClassName	none	schedule priority so the vault is not evicted first
disablePodDisruptionBudget	false	turn off the PDB
resources	none	container requests/limits

Put every secret (DEZHAN_VAULT_KEY, DEZHAN_SECRET, DEZHAN_ADMIN_TOKEN, DEZHAN_APPROVERS) in the referenced Secret. Nothing sensitive belongs in the CR.

Resilience

• A vault is a single writer over durable storage: one-replica StatefulSet with a ReadWriteOnce volume. Do not scale it; the immutability and audit-chain guarantees assume one writer. Cross-node durability and failover come from the underlying StorageClass (run on Ceph/cloud-disk/Longhorn so the PV reattaches).
• Pod runs non-root, read-only root filesystem, all capabilities dropped, seccomp=RuntimeDefault; only /data is writable.
• A PodDisruptionBudget (minAvailable=1) stops drains/upgrades from evicting the writer without a deliberate eviction. The StatefulSet retains its PVC on delete and scale, so data outlives the workload object. A startup probe tolerates slow journal replay; termination grace lets it flush.
• Online expansion: raise spec.storage and the operator grows the PVC in place (the StorageClass must allow expansion). Shrinking is never attempted.
• Reconciles are level-based and idempotent (no hot update loop); owned objects are watched, so a deleted Service/StatefulSet/PDB is recreated. The operator runs two replicas behind leader election with its own PDB.

Use as a Kubernetes volume (CSI)

The dezhan CSI driver turns a vault into a StorageClass: each PVC becomes a bucket, mounted on the node via mountpoint-s3. Best for write-once/append/archival workloads (it matches WORM); not for random-write volumes such as databases. For block/RWX-database storage, run dezhan on Ceph/Longhorn instead.

# edit deploy/csi/storageclass.yaml (endpoint) and secret-example.yaml (creds) first
kubectl apply -f deploy/csi/

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: archive
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: dezhan
  resources:
    requests:
      storage: 100Gi        # advisory; object storage is not pre-allocated

The controller's CreateVolume/DeleteVolume provision and remove a bucket via the dezhan S3 API; the node plugin's NodePublishVolume mounts it with mount-s3. Credentials come from the StorageClass-referenced Secret, never the image.

Volume snapshots are supported (deploy/csi/volumesnapshotclass.yaml): a VolumeSnapshot server-side-copies the bucket into an immutable snapshot bucket, and a PVC with that snapshot as its dataSource restores it.

Observability

/metrics is Prometheus format; the operator annotates each vault Service for scraping. Apply the ServiceMonitor, Grafana dashboard, alerts, and an OTel Collector bridge with kubectl apply -f operator/config/observability/.

Benchmarks

S3 throughput across a 3-node k3s cluster and an on-prem VM, with MinIO as a reference (Longhorn = dezhan on a Longhorn PV):

dezhan trades write throughput for durability and integrity (every write is fsync'd, erasure-coded, and encrypted), so it is much slower on PUT than MinIO but stays within a small factor on GET. Harness, raw results, and tables are in bench/ (bench/results/COMPARISON.md).

How it works

The retention state machine, clock-integrity guard, append-only audit chain, and erasure coding are written in SPARK and machine-checked by gnatprove (325 checks, 0 unproved); the cryptography (SHA-256/512, ChaCha20, HMAC, Ed25519) is in-tree with no external dependency. See docs/NOTES.md for design notes and current limitations.

Development

gprbuild -P dezhan.gpr           # build server, CLI, verifier
sh scripts/test.sh               # unit tests
sh scripts/prove.sh              # SPARK proof gate
sh scripts/coverage.sh           # trusted-core line coverage
( cd operator && go build ./... && go test ./... )
( cd csi && go build ./... && go test ./... )

Licensed under Apache-2.0.