Skip to content

Feat/quota config: Admins can now configure quotas#305

Open
St4NNi wants to merge 35 commits into
mainfrom
feat/quota-config
Open

Feat/quota config: Admins can now configure quotas#305
St4NNi wants to merge 35 commits into
mainfrom
feat/quota-config

Conversation

@St4NNi

@St4NNi St4NNi commented Jul 5, 2026

Copy link
Copy Markdown
Member

This PR extends the usage accounting feature: quota configuration as realm-replicated admin policy with admin endpoints, distribution of per-node usage counters over the sync layer with realm-wide aggregation, and a hard per-group byte quota gate on the S3 write paths. Includes the delta-test suite required by the issue.

Changes

  • Adds max_devices_per_user to QuotaConfig as stored configuration for the device cap.
  • Adds a RealmConfigQuotaSet admin document operation with reducer and overlay materialization so quota changes replicate, and fixes peers resetting quota to defaults when rebuilding realm config from reducer state.
  • Adds SetRealmQuotaOperation validating the warn threshold (1-100), global and per-override grace factors (>= 100), and duplicate group or user overrides.
  • Adds admin-gated PUT /api/v1/info/realm/quota (realm admin on a management node, replace semantics) and exposes quota in GET /api/v1/info/realm.
  • Materializes locally written quota from reducer state so conflicted quota paths behave identically on the local write and replicated overlay paths.
  • Adds NodeUsageSnapshot documents in a new usage_node_stats keyspace, keyed per node for global counters and group-first for per-group counters.
  • Adds DocumentSyncTarget::NodeUsage replicating snapshots over one shared realm-scoped topic.
  • Publishes snapshots through a debounced single-flight task: generation-stamped dirty markers written inside the counter transactions, a 2s debounce with a 5s durable rearm poll, and a full publish at startup after counter rebuild.
  • Clears dirty markers only after successful replication using compare-delete so concurrent writes and failed publishes never lose their retry signal.
  • Publishes zero snapshots for groups whose counters were pruned so stale remote per-group data converges.
  • Validates snapshot ingest cryptographically: only the keyholder of a node id can publish snapshots stored under that node's keys, and forged or malformed events of any kind are skipped without wedging the reconcile cursor.
  • Maintains a summed realm usage cache refreshed from all inbound and outbound reconcile handlers.
  • Extends GET /api/v1/info/usage and GET /api/v1/groups/{id}/usage with realm-wide totals; the realm section requires authentication while node-local fields keep their existing behavior.
  • Adds an embeddable QuotaGate enforcing per-group byte quotas inside the object put and multipart completion write transactions, summing live local counters and remote group snapshots.
  • Resolves the effective ceiling at the S3 surface as group override or default quota times the grace factor; no configured quota means no gate, and config load failures fail closed.
  • Rejects only writes that would exceed the grace ceiling with S3 403 QuotaExceeded; deletes, aborts, and zero-length writes are never gated.
  • Cleans up rejected writes: object put aborts the transaction and deletes the orphaned blob, and multipart completion aborts the open finalize transaction before resetting the upload, which also closes a pre-existing transaction leak on finalize errors.
  • Adds delta tests for every write path (overwrite, delete markers, put over a delete marker, aborted multipart, content dedup) with a rebuild oracle asserting incremental counters match a full recount after every scenario.
  • Adds replication tests for cross-node aggregation, counter-neutral ingest of counter-bearing snapshots, and steady-state publishing without restart.
  • Adds NodeUsage targets to the doctor explorer json output.

GitHub Issues

  • Completes [feat] Usage accounting and quota configuration #249: quota configuration lives in the realm config as replicated admin policy with API endpoints, per-node and per-group counters distribute over the sync layer with a summed local cache, and the issue's test concept (per-write-path delta tests, explicit replication-ingest neutrality) is implemented.
  • Partially addresses [feat] Quota enforcement and reporting #250 by landing the hard enforcement gate that rejects writes above the grace ceiling. This PR does not add warning notifications, hysteresis, usage history snapshots, portal UI, or the periodic recount task.
  • Device cap enforcement is intentionally out: no flow creates user-device nodes yet and RealmNode carries no ownership, so enforcement is blocked on [feat] User node kind and device enrollment #271. The config field is stored for it.

Notes

  • Counters on remote nodes are stale by design (roughly 2-7s publish latency); the grace headroom is the documented budget for that staleness.
  • Snapshot admission binds the publishing node's identity, not group membership: a node can still misreport its own counters, which now has enforcement consequences. Follow-up for [feat] Quota enforcement and reporting #250/[feat] Balancer and dead node handling #265.
  • Snapshots of departed nodes linger in usage_node_stats and keep counting toward realm totals; cleanup belongs to dead-node handling in [feat] Balancer and dead node handling #265.
  • Known follow-ups: the gate's in-transaction remote scan adds a commit-conflict source on hot puts, per-request realm config reads could take a short-TTL cache, and the quota overlay clause exists in two places and should get one canonical helper in core.

St4NNi added 30 commits July 4, 2026 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant