feat(worker): add disk free-space ratio protection (free_ratio)

[gangump82]

What & why

Adds a per-disk free-space ratio floor (free_ratio) so a worker stops writing to a storage directory once its free-space ratio drops below a threshold. Closes #1643.

curvine currently only supports a fixed-byte per-dir reservation (dir_reserved), which cannot adapt to disks shared with other consumers or to mixed disk sizes, and provides no early-warning signal before the disk fills. free_ratio (default 0.0, disabled, backward compatible) gates VfsDir::available() to 0 when raw_free_ratio() < free_ratio, covering both FS mode (statvfs available/total) and SPDK raw-device mode ((capacity − used)/capacity).

Design

• One lever: VfsDir::available() returns 0 below the threshold. All write-admission flows pass through it — worker can_allocate, rewrite precheck, and heartbeat → master allocatable_available → chooser — so worker-side rejection and master-side stop-allocation apply with no master code change. Eviction is untouched (orthogonal metadata-entry quota).
• Config: WorkerConf.free_ratio: f64 (default 0.0; range [0.0, 1.0); out-of-range clamped with a warning).
• Metrics: disk_free_ratio per-disk GaugeVec (labeled dir_id/dir_path/storage_type; basis points, since the crate's gauges are i64-only) + disk_full_rejected_writes Counter, instrumented at the write-handler layer via stable error-message matching so curvine-storage-local stays decoupled from curvine-metrics.
• Example config entry + unit tests (FS/SPDK gate, raw_free_ratio, disabled-by-default, config parse/clamp).

Cluster verification

Tested on an internal 3-node dev cluster with a release build of this branch running on one worker node only (master/journal/fuse untouched). See #1643 for the full result matrix. Summary: with free_ratio=0.995 above the disk's actual 99.05% free, available correctly dropped to 0 (gauge kept reporting the real pre-guard ratio), a FUSE write was routed to a healthy worker (master excluded the gated one), and free_ratio=0.0 restored available immediately. The production worker was restored to the original binary afterward.

Backward compatibility

Default 0.0 disables the guard → available() behaves identically to today; configs without free_ratio get 0.0 via existing #[serde(default)].

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a configurable per-disk free-space ratio floor (worker.free_ratio) so workers stop admitting new writes to a storage directory once the underlying device’s free ratio drops below the threshold, and exposes supporting metrics/config examples.

Changes:

• Introduces WorkerConf.free_ratio (default 0.0) and documents it in the example cluster TOML.
• Adds a free_ratio guard in VfsDir::available() plus raw_free_ratio() computation and plumbing from config → dataset → dir.
• Exposes per-dir free ratio metrics (disk_free_ratio) and increments a rejection counter on write admission failures.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
curvine-worker/src/worker/worker_metrics.rs	Adds disk_free_ratio GaugeVec and disk_full_rejected_writes Counter; publishes per-dir ratios.
curvine-worker/src/worker/handler/write_handler.rs	Increments rejection counter when storage admission fails with “Not enough space” errors.
curvine-docker/deploy/example/conf/curvine-cluster.toml	Documents free_ratio configuration and usage.
crates/common/curvine-config/src/worker_conf.rs	Adds free_ratio field with defaults and TOML parsing tests.
crates/adapters/curvine-storage-local/src/vfs_dir.rs	Adds free_ratio to VfsDir, guards available(), and implements raw_free_ratio().
crates/adapters/curvine-storage-local/src/vfs_dataset.rs	Clamps free_ratio from config, passes it into VfsDir, and reports per-dir ratios via dataset API + tests.
crates/adapters/curvine-storage-local/src/dataset.rs	Adds DirFreeRatio and extends the Dataset trait with dir_free_ratios().

Suppressed comments (1)

curvine-worker/src/worker/worker_metrics.rs:111

• The Prometheus help text for disk_full_rejected_writes currently says it is caused by dropping below free_ratio, but the increment logic is based on generic "Not enough space" admission errors. The help string should match what is actually being counted (insufficient capacity / admission rejection), otherwise dashboards will be misleading.

            disk_full_rejected_writes: m::new_counter(
                "disk_full_rejected_writes",
                "Writes rejected because a data dir dropped below free_ratio",
            )?,

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[szbr00]

Required: Gating available() to 0 also fails in-place rewrite of
already-placed FileLayout blocks. BlockStore::open_block ->
reserve_file_open / Dataset::open_block both reject when
required_bytes > dir.available(). After the master stops new
allocation, a client overwrite of a file whose block lives on the
gated dir still has to come back to this worker; it now gets a space
error even when the disk still has gigabytes free (e.g. 9% free with
free_ratio=0.1) and the rewrite is not attracting new data.
FileLayout rewrite always charges a full staging copy
(preserves_committed_on_write, then fs::copy), so available()==0
is a hard write-freeze, not just "stop allocating". Cluster
verification only covered a new FUSE write routed to another worker.

Split the lever: keep the guard on can_allocate and on the
heartbeat StorageInfo.available used by the master chooser, and let
rewrite prechecks use ungated remaining bytes (so an existing block
can still be updated while the 10% headroom is reserved for other
apps and for the staging copy). Add a test: finalize a FileLayout
block, set free_ratio above raw_free_ratio, rewrite the same
size, and assert the open succeeds. If write-freeze of existing
files is the intended product behavior, document that on
WorkerConf.free_ratio and assert the rejection instead — do not
leave it implicit.