feat(configs+queue): managed-config server with token auth; queue_output bridge - #405
Merged
Conversation
Member
Author
Merge-readiness update (
|
The portable server side of the standard managed-config channel was developed on the otel branch but the squash merges (#402/#403) only carried the open-source pipeline pieces — the package itself never landed on main. Bringing it (and the register double-read fix fc65f86 developed on top of it) over: - POST /instance/_register: bare Instance or {client:{...}} payloads, read-once body handling (the double-read broke every bare-Instance registration with 'invalid Read on closed Body'), per-instance token minting in the response - POST /configs/_sync: heartbeat + hash fast-path + version diff; instance-token validation when minted - POST /instance/_exchange_token: rotation with 1h grace - sha256-at-rest tokens, constant-time compares, open dev mode warns Consumers: LogPilot (ingestion center) and Gateway (cascading tiers).
Two-stage pipelines: process on one queue, sink (e.g. bulk_indexing, itself consumer-shaped) from another. Push failures return an error so the consumer leaves the offset uncommitted (at-least-once). Used by LogPilot's stream compiler for easysearch sinks.
Resolves a processor constructor by its registered bare name — lets hosts build ad-hoc chains from spec entries (LogPilot's dry-run replay is the first consumer).
…tion The clone processor (enterprise, Graylog clone_message parity) deposits cloned records via AppendClone; for_each materializes them as extra batch members after each record's sub-chain (shared offset, drop markers honored) and publishes the extended batch back to the context.
sqlite Get returns (false, ErrNotFound) for missing rows and GetV2 propagates it verbatim; upsertInstance returned the error, so EVERY fresh registration (and its sync heartbeat) failed with 500 'record not found' and the instance never landed. Not-found is the normal create path — treat it as exists=false; only real backend errors propagate.
Adopt the framework's own access-token management (console_framework lineage) instead of growing a parallel scheme: - client: the full managed client (545-line console version) — bootstrap token recovery from keystore/config on 401, post-register hooks (token exchange runs immediately after a successful register), atomic re-register, X-API-Token/Bearer auth precedence - common: token.go keystore helpers + InstanceRegisterRequest wrapper + config/domain updates - model: const API_TOKEN header key; Instance.AccessToken persists the agent's self-minted API token at registration (both payload shapes) - config: ManagerConfig.AccessToken (bootstrap value; the exchanged token lands in the keystore) The LogPilot ingestion detail panel proxies agent /pipeline/tasks/ using this stored token — closing the loop end-to-end.
…cked the minted one The console client authenticates manager requests with X-API-Token = the agent's SELF-minted API token (registered in Instance.AccessToken), while the sync handler only validated the server-minted InstanceToken (whose plaintext the agent never learns without the exchange endpoint). Every sync after registration failed 401, then the recovery path errored 'managed bootstrap access token is missing' (nothing to recover from — the agent has no bootstrap token configured, nor should it need one against an open/static-gated server). Sync now accepts EITHER credential: the minted InstanceToken (Bearer) or the registered self API token (constant-time compare against Instance.AccessToken). Register-with-self-token + sync-with-self-token works out of the box; exchange rotates to minted tokens later.
Simplified enrollment for agents/gateways with a management-side approval step: - register: new instances land with Status=pending — visible in the management UI (heartbeat flows) but receive NO credentials and NO configs. The register response carries approved=false. - sync: pending instances get an empty config set; heartbeats still refresh the record so the UI shows them live while awaiting approval. - POST /instance/:id/_approve (management, token-gated): flips status to approved and mints the per-instance token. The instance picks it up automatically — the client keeps re-registering while unapproved (no local registration marker is written until approved) and captures manager_token from the register response into its keystore. - Once approved, sync auth (minted InstanceToken OR registered self token) and config delivery proceed as before. Flow: agent up → appears as 待准入 in the ingestion center → admin clicks Approve → next register cycle (30s) receives the credential → secure channel established, configs start flowing.
…onal Two admission-flow regressions: - sync enforced credential checks on every instance that held a minted token — including PENDING ones, whose credential pairing isn't established yet (and legacy rows with empty status). Heartbeat sync from a pending instance 401'd in a loop and the UI never saw it. Enforcement now applies to APPROVED instances only; pending sync carries nothing sensitive (empty config set) and heartbeat visibility is the point. Empty (legacy) status counts as pending. - the 401 recovery path demanded a bootstrap token and failed hard when none was configured — wrong for admission-mode servers where approval (not a pre-shared bootstrap) is the gate. Bootstrap restore is now best-effort: plain re-registration is the correct recovery when no static token exists.
…gister rate limiting Closes the publicly-reachable register surface: - EnrollmentToken: sha256-at-rest, max_uses (1 = one-time), TTL, revocation; plaintext shown exactly once at generation - POST/GET /instance/_enrollment_tokens, DELETE .../:id (admin, token-gated) — mint, list (masked + status), revoke - register validates and CONSUMES the ticket when configs.server.enrollment.required: true (X-Enrollment-Token header or enrollment_token body field); invalid/expired/exhausted → 403 before any record is written - register rate limit per client IP (default 10/min fixed window, register_rate_limit config; proxy headers honored) - client: configs.enrollment_token config, sent on register Deployment model: operator mints a ticket in the UI → embeds it in the agent's config out-of-band → agent redeems it at registration → admission flow (pending → approve) takes over. Layered: network segmentation (ops) → enrollment ticket → rate limit → manual approval → per-instance credentials → optional mTLS.
… line
Repeatable -e pairs are applied to the process environment before the
config loads, feeding the $[[env.KEY]] template expansion (OS env wins
over the YAML env: section). Deployments can now parameterize any
templated setting without editing config files:
agent -e MANAGED=true \
-e REMOTE_CONFIG_SERVERS=http://logpilot:29000 \
-e ENROLLMENT_TOKEN=et-abc123...
Verified A/B: -e MANAGED=false skips config-manager registration
entirely; without it the agent registers as usual.
Register's re-authentication for existing instances only accepted the minted InstanceToken or a static token — the client actually presents X-API-Token = its self-minted API token (Instance.AccessToken), which sync already accepts (matchesRegisteredAccessToken) but register did not. Every re-registration 401'd in a loop: boot → unauthorized → clear state → re-register → unauthorized → ... Accept the registered self token on re-register too, mirroring sync.
…ken manager Approve/register/exchange mint the credential via access_token.CreateAPIToken instead of the custom InstanceToken scheme: - standard storage (ORM record + KV fast lookup), standard revocation and token-management UI/API - instance binding via token Data.instance_id — a manager token can never authenticate a different instance (checked on every sync) - permissions attachable for scoped manager capabilities later - type "managed_instance" distinguishes them in the token list Sync/re-register accept: the standard manager token, or the agent's registered self API token (pre-exchange), or (re-register only) a static admin token. The custom InstanceToken path is retired. Also carries the interrupted enrollment fix from earlier: validate the ticket without consuming; burn a use ONLY when registration actually creates the instance (pending agents re-register every cycle and were exhausting limited-use tickets).
upsertInstance defaulted the incoming payload's empty Status to pending and saved it — every sync/re-register overwrote an approved instance's status, so after clicking Approve the agent's next register response said approved=false forever and it never received its manager token. Status is a server-owned admission field: on update, preserve the stored value; only fresh registrations default to pending.
The approved flag in the register response came from the incoming instance payload — agents never send a status, so even an approved agent's register response said approved=false and it looped in the 'waiting for admin approval' state forever despite the DB saying approved. Read the server-owned status back from the stored record.
The exchange endpoint (self token → manager token) only accepted the retired InstanceToken or a static token — so the agent's post-register exchange with its self token (or the already-issued manager token) 401'd, failing the register hook and retriggering the recovery loop even though sync itself now succeeds. Accept the standard manager token and the registered self API token, mirroring sync/re-register.
GetInstanceInfo always published APIConfig.GetEndpoint() — the default :2900 (with skip-if-occupied), which most deployments never actually serve. Agents/gateways disable the API port (api.enabled: false) and serve on the web port, so every registered instance advertised the same wrong http://host:2900, breaking manager→instance callbacks (detail-panel pipeline tasks, proxying). When the API server is disabled, advertise the web address instead (schema from the web TLS config).
Managed agents are typically behind NAT/firewall: they can only dial OUT to the manager, and the advertised endpoint is usually NOT reachable. The reverse channel (core/api/websocket/reverse, #401) is the designed answer — the agent connects to the manager's /ws endpoint carrying its instance ID, HELLOs, and the manager then ProxyRequests down that connection. Server side wired into the configs server: - websocket connect/disconnect callbacks: peer header → instance must exist + be approved + present a valid credential (manager token or registered self token) → pending session - HELLO/RESPONSE commands feed the SessionManager - exported ReverseProxyRequest/ReverseIsConnected for consumers (LogPilot's instance detail is the first) Agent side (agent repo): internal/reverse — a lean self-contained client that dials the configs servers' /ws, handshakes, and executes proxied requests as loopback HTTP calls against the agent's own web port (authenticated with its API token). No console-lineage API-router plumbing needed.
The ticket check ran unconditionally when enrollment.required was on — an approved agent re-registering after restart (with its persistent manager/self credential) got 403 'invalid enrollment token' because the one-day ticket from its first enrollment had long expired. Tickets are for NEW registrations only: an existing instance that presents a valid manager token or registered self token re-registers without one. This preserves the credential lifecycle: ticket → first register → approve → manager token (long-lived) → re-registers authenticated by the token.
…ssTokenKeystoreKey The token is minted for any Framework-based instance registering with the console (Agent, Gateway, third-party apps), not just Agent. Rename the keystore key to instance_access_token and drop the agent/gateway whitelist in SupportsManagedAccessToken so any named application qualifies.
…/Gateway use configs/server + reverseclient: - ListTokens returns []*AccessToken (value copies of the lock-bearing struct were flagged by vet in server and downstream consumers) - manager token listing iterates by pointer elastic/indexing_merge: - normalizeDataStreamDoc: otel envelope -> data stream doc (payload promoted to top level, @timestamp derived with explicit priority, metadata file/log_type/log_kind labels kept); only for write_op_type create — covered by new unit tests processors/grok: - ECS-style dotted capture names (process.pid, safepoint.name) compile: regex group names sanitized, extraction uses the real field name - singular "pattern" (string) accepted as a one-element patterns list easysearch: - cluster CRUD PostCreate/PostUpdate register the live ES client immediately (enables manager-pushed sink clusters on gateways) core/api: - StartWeb skips duplicate /ws registration when embedding_api already mounted it (Go 1.22+ ServeMux panics on duplicate patterns) Verified: go vet clean on touched packages; tests green (configs, indexing_merge, grok, reverseclient) incl. -race; LogPilot/Gateway/Agent build and run against this branch end-to-end.
medcl
force-pushed
the
configs-server-and-queue-output
branch
from
August 24, 2026 13:58
46c642b to
488d169
Compare
SteveLauC
approved these changes
Aug 25, 2026
A pending (not yet approved) instance synced against an empty assigned set, so the diff reported every local managed file as Deleted — wiping config files and stopping running pipelines (e.g. after an instance re-registered under a new identity and was momentarily pending). Pending syncs now short-circuit to Changed:false; the diff resumes once the instance is approved.
The loopback executor always rewrote the binding host to 127.0.0.1. For an instance whose web port is pinned to a specific interface IP (e.g. a second gateway on a LAN address), nothing listens on the loopback rewrite and every proxied request fails. Only wildcard bindings (0.0.0.0/::) are rewritten now.
Member
Author
|
Added two fixes found in follow-up live testing of this stack (both confined to
Both verified on a live two-gateway/two-agent isolation setup; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two pieces developed on the otel line that the squash merges (#402/#403) did not carry to main — LogPilot's ingestion center and stream compiler depend on both.
configs/server (protocol server, was fc65f86 lineage)
POST /instance/_register: bare Instance + {client:{...}} payloads, read-once body handling — the original double-read broke every bare-Instance registration with 'invalid Read on closed Body' (deployed agents send that shape); mints a per-instance token in the responsePOST /configs/_sync: heartbeat + hash fast-path + version diff; instance-token validationPOST /instance/_exchange_token: rotation with 1h grace; sha256-at-rest, constant-time compares; open dev mode warns loudlyplugins/queue/queue_output
Chain-tail batch→queue bridge enabling two-stage pipelines (process on one queue, bulk_indexing from another). Push failure → error → offset uncommitted (at-least-once).
Verified