diff --git a/docs/deployment/clustering.mdx b/docs/deployment/clustering.mdx index 5382eb2..f0f73b3 100644 --- a/docs/deployment/clustering.mdx +++ b/docs/deployment/clustering.mdx @@ -31,6 +31,25 @@ FLO_CLUSTER_SEEDS=10.0.1.10:9000,10.0.1.11:9000 FLO_NODE_ID=1 ``` +### Ports + +A clustered node binds an additional Raft port for inter-node traffic: + +| Port | Purpose | +|------|---------| +| `listen_port` | Client wire protocol | +| `listen_port + 500` | Raft RPC between nodes (9500 by default) | +| `listen_port + 600` | Gossip, only when `gossip_port` is set | + +The Raft port is bound **only when the node can actually have peers** — when +any of `[cluster] enabled = true`, `seeds`, an explicit `raft_port`, or +`replication_factor > 1` is configured. A single-node server leaves it unbound, +so it is not a port operators need to declare to an orchestrator. The startup +banner reports either the bound port or `not listening (single-node)`. + +Raft and gossip ports carry unauthenticated inter-node traffic and must not be +exposed outside the cluster's private network. + ### Bootstrap The first node in a cluster bootstraps automatically when no seeds respond. Subsequent nodes join by contacting any existing member. diff --git a/docs/deployment/docker.mdx b/docs/deployment/docker.mdx index 538d1b7..1a28b3e 100644 --- a/docs/deployment/docker.mdx +++ b/docs/deployment/docker.mdx @@ -14,6 +14,7 @@ docker pull ghcr.io/floruntime/flo:latest ```bash docker run -d \ --name flo \ + --security-opt seccomp=unconfined \ -p 9000:9000 \ -p 9001:9001 \ -p 9002:9002 \ @@ -21,11 +22,60 @@ docker run -d \ ghcr.io/floruntime/flo:latest ``` +:::note +`--security-opt seccomp=unconfined` is required on Linux hosts. Flo's event +loop uses io_uring, which Docker's default seccomp profile blocks — without it +the server starts, logs `topology manifest created`, and then exits. See +[io_uring and seccomp](#io-uring-and-seccomp) for the details and for a +narrower profile. +::: + | Port | Purpose | |------|---------| | 9000 | Binary wire protocol (client traffic) | -| 9001 | Prometheus metrics | -| 9002 | Dashboard REST API | +| 9001 | Prometheus metrics — only when `[metrics] enabled = true` | +| 9002 | Dashboard REST API — only when `[dashboard] enabled = true` | + +Flo does **not** listen on any other port in a single-node deployment. In +particular the Raft port (`listen_port + 500`, so 9500) is bound only when the +node is actually clustered — see [Clustering](/deployment/clustering). + +## io_uring and seccomp + +On Linux, Flo's event loop is built on io_uring. Docker's default seccomp +profile does not allow the `io_uring_setup`, `io_uring_enter` and +`io_uring_register` syscalls, so under that profile Flo cannot start: + +``` +INF topology manifest created: shards=1 partitions=0 path=/data +ERR io_uring is unavailable (PermissionDenied). ... +Error starting runtime: error.PermissionDenied +``` + +The simplest fix is to run unconfined: + +```bash +--security-opt seccomp=unconfined +``` + +If you would rather not drop the whole profile, copy Docker's +[default profile](https://github.com/moby/moby/blob/master/profiles/seccomp/default.json) +and add the three syscalls to its allowlist: + +```json +{ + "names": ["io_uring_setup", "io_uring_enter", "io_uring_register"], + "action": "SCMP_ACT_ALLOW" +} +``` + +Then run with `--security-opt seccomp=/path/to/flo-seccomp.json`. + +:::note +io_uring requires **Linux 5.1 or newer**. On older kernels Flo cannot start at +all, and no seccomp profile will help. Kubernetes, Podman and containerd apply +their own seccomp defaults — the same allowance is needed there. +::: ## Docker Compose — Single Node @@ -34,6 +84,8 @@ version: "3.8" services: flo: image: ghcr.io/floruntime/flo:latest + security_opt: + - seccomp:unconfined ports: - "9000:9000" - "9001:9001" @@ -56,6 +108,8 @@ version: "3.8" services: flo-1: image: ghcr.io/floruntime/flo:latest + security_opt: + - seccomp:unconfined hostname: flo-1 ports: - "9000:9000" @@ -71,6 +125,8 @@ services: flo-2: image: ghcr.io/floruntime/flo:latest + security_opt: + - seccomp:unconfined hostname: flo-2 ports: - "9010:9000" @@ -86,6 +142,8 @@ services: flo-3: image: ghcr.io/floruntime/flo:latest + security_opt: + - seccomp:unconfined hostname: flo-3 ports: - "9020:9000" @@ -122,6 +180,8 @@ All `flo.toml` settings can be overridden with `FLO_` prefixed environment varia ## Health Check +For a **liveness** probe inside the container, the CLI is enough: + ```yaml healthcheck: test: ["CMD", "flo", "server", "status"] @@ -130,6 +190,25 @@ healthcheck: retries: 3 ``` +For an orchestrator **readiness** probe that comes from outside the container, +use `GET /health` on the dashboard port (see +[REST API](/reference/rest-api#cluster-metrics)) — and set the dashboard's bind +address, because it defaults to localhost only: + +```toml +[dashboard] +enabled = true +bind = "0.0.0.0" +``` + +:::note +`[dashboard] bind` defaults to `127.0.0.1`. A probe originating outside the +container cannot reach a listener bound to loopback, so leaving the default in +an orchestrated deployment presents as a service that never becomes ready — +which reads like a broken health check rather than a bind address. Bind to +`0.0.0.0` only where the dashboard port is not publicly routable. +::: + ## Volumes and Data Persistence Always mount a volume at `/data` to persist data across container restarts: diff --git a/docs/getting-started/configuration.mdx b/docs/getting-started/configuration.mdx index 8b23d39..6672391 100644 --- a/docs/getting-started/configuration.mdx +++ b/docs/getting-started/configuration.mdx @@ -48,11 +48,28 @@ hot_buffer_capacity = 67108864 # Per-partition ring buffer size in bytes (64 M Durability modes: -| Mode | Behaviour | -|------|-----------| -| `sync` | `fsync` after every write — strongest guarantee, lowest throughput | -| `async_flush` | Background flush every ~1 ms — default, highest throughput | -| `ephemeral` | Skip WAL entirely — for caches or temporary data | +| Mode | Behaviour | At risk on abrupt termination | +|------|-----------|-------------------------------| +| `sync` | Segment written and `fsync`ed before the write is acknowledged | Nothing | +| `async_flush` | Segments written and `fsync`ed by a background task, **at most once per second** — the default | Up to ~1 second of acknowledged writes | +| `ephemeral` | Never written to disk | Everything | + +**What `async_flush` risks, concretely.** An acknowledged write lives only in +memory until the next segment flush, which runs at most once per second. If the +process is killed (`SIGKILL`, OOM kill, power loss, host failure) you can lose +up to roughly the last second of acknowledged writes. Everything older is on +disk, `fsync`ed and published by atomic rename, so the store is never left +torn or half-written — the loss is a clean tail truncation, not corruption. + +A **graceful** shutdown (`SIGTERM`, `docker stop`, `flo server stop`) flushes +before exiting, so a normal restart loses nothing. The one-second window only +applies when the process dies without running its shutdown path. + +:::note +This window is **not** governed by `hot_flush_seconds`. That setting controls +hot → warm tier migration, which is a storage-layout concern and does not +affect what survives a crash. +::: ### `[logging]` @@ -88,9 +105,12 @@ pong_timeout_ms = 10000 # Close connection if no pong within this time [metrics] enabled = true # port = 0 # 0 = auto (listen_port + 1), so 9001 by default -# bind = "0.0.0.0" +# bind = "127.0.0.1" # Localhost only by default ``` +Binds only when `enabled = true`. For a Prometheus server scraping from outside +the container, set `bind = "0.0.0.0"` — the default is loopback-only. + ### `[dashboard]` ```toml @@ -101,6 +121,14 @@ enabled = true # cors_origins = "http://localhost:5173" ``` +:::note +`bind` defaults to `127.0.0.1`. In an orchestrated deployment a readiness probe +originating outside the container cannot reach a loopback listener, so the +default presents as a service that never becomes ready. Set `bind = "0.0.0.0"` +where the port is not publicly routable. See +[Docker → Health Check](/deployment/docker#health-check). +::: + ### `[cluster]` ```toml @@ -115,6 +143,12 @@ enabled = false # heartbeat_interval_ms = 50 ``` +The Raft port is bound only when this node can actually have peers — that is, +when any of `enabled = true`, `seeds`, an explicit `raft_port`, or +`replication_factor > 1` is set. A plain single-node server does **not** listen +on `listen_port + 500`, so there is no undeclared port to account for. The +startup banner reports which of the two applies. + ### `[cold_storage]` ```toml diff --git a/docs/reference/rest-api.mdx b/docs/reference/rest-api.mdx index d61408a..11009ec 100644 --- a/docs/reference/rest-api.mdx +++ b/docs/reference/rest-api.mdx @@ -88,7 +88,32 @@ Aggregated metrics in JSON (Prometheus metrics are exposed separately on port 90 ### `GET /health` -Always-public liveness check (served at the root, not under `/api/v1`). +Liveness and readiness check. Served at the root, not under `/api/v1`, and +always public — it is never gated by the admin token, so it is safe to point an +orchestrator probe at it without provisioning credentials. + +```json +{ "status": "ok" } +``` + +A `200` means the dashboard server is up and serving. Any other status, or a +refused connection, means the node is not ready. + +**Stability:** `GET /health` is a stable endpoint. The path, its public status, +and the `200` + `"status": "ok"` contract will not change within a major +version. Fields may be *added* to the object, so parse it leniently rather than +matching the body exactly — checking the HTTP status alone is the most durable +probe. + +:::note +This is served on the **dashboard** port (`listen_port + 2`, so 9002 by +default), which requires `[dashboard] enabled = true`. `[dashboard] bind` +defaults to `127.0.0.1`, so a probe from outside the container needs +`bind = "0.0.0.0"` — see [Docker](/deployment/docker#health-check). + +The Prometheus exporter serves its own separate `/health` on `listen_port + 1`, +which additionally reports the shard count. +::: ---