feat: add privacy-preserving product telemetry

[cwilhit]

What this does

This adds an OSS-visible and friendly product telemetry system for Faros SaaS deployments:

• A checked-in, provider-owned event catalog with centrally defined metrics, validation, generated Go registry data, and generated catalog documentation.
• A default-off hub runtime that validates catalog events, pseudonymizes identifiers with HMAC, queues and batches delivery with bounded retries/shutdown, and emits CloudEvents to a configured receiver.
• Authenticated provider ingestion through workspace-scoped Kubernetes TokenReview, plus a reusable provider SDK so future providers can emit cataloged events without coupling to a specific analytics vendor.
• Product event instrumentation for organization/workspace creation, provider enablement, App Studio project creation/preview readiness/publishing, Agents agent creation/run completion, and first edge readiness.
• A standalone receiver with strict decoding, PostgreSQL-backed idempotency and aggregate projections, retention/erasure support, TLS configuration, Helm packaging, and optional Grafana assets.
• OSS-facing documentation for the data contract, privacy model, configuration, operational behavior, and extension workflow.

Why

We need trustworthy activation and usage signals when Faros is operated as SaaS, while preserving the expectations of an OSS installation.

The implementation therefore makes telemetry an explicit SaaS operator choice: stock installations remain off, start no telemetry infrastructure, expose no provider ingestion route, and make no telemetry network calls. When enabled, only cataloged bounded fields are accepted; raw content is prohibited, identifiers are pseudonymized before transport, and the catalog remains reviewable in the OSS repository. Provider integrations target the Faros telemetry contract rather than a hosted analytics vendor, so the sink can evolve or be replaced independently.

Events covered

• organization_created
• workspace_created
• provider_enabled
• app_studio_project_created
• app_studio_preview_ready
• app_studio_project_published
• agents_agent_created
• agents_run_terminal
• edge_first_ready

Notable hardening included

Review and live verification uncovered and fixed several lifecycle issues along the instrumented paths:

• Provider enablement now records successful recovery from a partially completed binding/grant operation, while remaining idempotent on a fully enabled provider.
• Agents runs without configured model credentials now reach a durable terminal failure instead of remaining Pending.
• Agents creation and run terminalization use durable database claims/CAS semantics so retries, concurrent requests, and multiple replicas do not duplicate telemetry.
• Receiver delivery supports private CAs and enforces a secure Helm/TLS configuration in SaaS mode.

Verification

Automated checks

• make verify-telemetry
• Fresh focused root tests for the catalog/codegen, hub runtime, provider ingress, REST/KCP integration, receiver, and receiver command packages.
• provider-sdk/telemetry: full focused test suite.
• providers/agents: full tests, race tests, and go vet ./....
• Live PostgreSQL integration tests for receiver/Agents idempotency, including exclusive run finalization.
• git diff --check.

Isolated positive E2E cluster

An independent kind/Tilt cluster was used so no other development cluster was touched. All nine events were exercised through their real Faros product paths and observed at the receiver and aggregate projection layers:

• Organization and workspace creation returned 201 and produced one event/metric each.
• Six platform providers were enabled through the real API; each produced the expected metric, while re-enabling Edges remained idempotent.
• App Studio project creation, preview authorization/readiness probing, and production promotion produced their three expected events. The App Studio API/state transitions and actual readiness HTTP probe were real; downstream Code/Infrastructure objects were synthetic fixtures because the isolated cluster did not include a KRO runtime.
• Agents creation produced exactly one event across a repeated create request. A no-credential run durably reached Failed; retrying reused the same run and did not increment telemetry.
• An edge completed join-token exchange, tunnel connection, and Ready; reconnecting did not duplicate the first-ready event.
• Receiver projections and activation funnels matched expected counts. Hub telemetry reported no dropped events.
• Database inspection confirmed that raw org, workspace, project, agent, run, edge identifiers, prompts, errors, and preview URLs were absent.

Fresh default-off E2E cluster

A second clean kind/Tilt installation was deployed with all telemetry values left at their defaults:

• No telemetry receiver, PostgreSQL workload, telemetry secret, or telemetry CA resource was created.
• The hub rendered only --telemetry-mode=off; it had no telemetry endpoint or telemetry credential configuration.
• Organization and workspace creation still succeeded and persisted normally.
• The provider telemetry ingress route returned 404.
• Telemetry metric samples remained zero before and after product activity, and no telemetry emissions appeared in logs.
• Agents, App Studio, and Edges provider charts rendered FAROS_PRODUCT_TELEMETRY_ENABLED=false.

[mjudeikis]

Just dont merge it yet :) I had some other idea how this can be done.
using k8s events and a generic backend (not dependent on cloud infra). Like just trow it all to clickhouse