diff --git a/.env.production.example b/.env.production.example index 9b67090..17ca6a2 100644 --- a/.env.production.example +++ b/.env.production.example @@ -15,6 +15,9 @@ RUN_MIGRATIONS=true PORT=8000 DATABASE_MAX_CONNS=20 UPDATE_CHECK_RETENTION_DAYS=90 +CLIENT_CHECKIN_RETENTION_DAYS=7 +AUDIT_LOG_RETENTION_DAYS=180 +AUDIT_METADATA_RETENTION_DAYS=90 # Shared high-entropy bearer token for the backend and delta worker. WORKER_TOKEN=replace-with-a-long-random-value diff --git a/AGENTS.md b/AGENTS.md index c6389f7..63fd98a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,13 +1 @@ -## graphify - -This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships. - -When the user types `/graphify`, invoke the `skill` tool with `skill: "graphify"` before doing anything else. - -Rules: -- For codebase questions, first run `graphify query ""` when graphify-out/graph.json exists. Use `graphify path "" ""` for relationships and `graphify explain ""` for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output. -- Dirty graphify-out/ files are expected after hooks or incremental updates; dirty graph files are not a reason to skip graphify. Only skip graphify if the task is about stale or incorrect graph output, or the user explicitly says not to use it. -- If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing. -- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context. -- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost). -- Never commit unless explicetly asked too +# Instructions diff --git a/CLIENT.md b/CLIENT.md index 0524114..403d14a 100644 --- a/CLIENT.md +++ b/CLIENT.md @@ -87,7 +87,7 @@ avoids expiry races. Deduplicate refreshes inside the process so concurrent features share one request. After transient failures, retry with exponential backoff and random jitter rather than on every foreground event. -``` +```http Authorization: Bearer ``` @@ -123,7 +123,7 @@ If the cached token has only recently expired (within the same seven-day window optional update check. Prefer it for scheduled background work because the server authorizes the license and activation only once. -``` +```http Authorization: Bearer ``` @@ -134,6 +134,7 @@ Authorization: Bearer "build": "140", "platform": "macos", "arch": "arm64", + "osVersion": "15.0", "channel": "stable", "clientId": "" } @@ -144,12 +145,14 @@ The response always contains `token`, `validUntil`, `updateChannels`, and `updateStatus`. `updateStatus` is `not_requested`, `ok`, or `unavailable`; an `ok` response also includes the normal update-check payload in `update`. -When `version` is present and the license is authorized, Clave also records a -best-effort client check-in for the Version Adoption dashboard. This happens -independently of update resolution, so the check-in still counts when no update -source is configured or the update provider is temporarily unavailable. Build, -platform, architecture, and OS version are optional diagnostics. Check-in -recording never changes the synchronization response or failure behavior. +Every authorized synchronization updates the activation's last-seen time. When +`version` is present, Clave also stores the current version, build, platform, +architecture, and OS version and records a check-in for the Version Adoption +dashboard. The update happens before update resolution, so the check-in still +counts when no update source is configured or the update provider is temporarily +unavailable. Build, platform, architecture, and OS version are optional. If +Clave can't persist this state, synchronization fails rather than returning +success with stale device data. --- @@ -282,7 +285,7 @@ resume with `Range`, verify the published SHA-256, then rename the completed file atomically. S3-backed artifacts redirect to short-lived presigned URLs and should use the same resume and checksum workflow. -``` +```http Authorization: Bearer ``` @@ -315,7 +318,7 @@ Use `name` as the `channel` value in update checks. If you're not giving users a Use your license JWT to check if a newer version is available. The server resolves which update backend to use based on the product's configuration. -``` +```http Authorization: Bearer ``` diff --git a/PRIVACY.md b/PRIVACY.md new file mode 100644 index 0000000..467bf2c --- /dev/null +++ b/PRIVACY.md @@ -0,0 +1,349 @@ +# Privacy and data inventory + +This page lists the data that Clave can receive, derive, store, transmit, or +show to an operator. It is a technical inventory for self-hosted deployments, +not legal advice or a complete privacy notice. + +The organization that deploys Clave usually decides why and how to process +personal data, which usually makes it the controller. Infrastructure, email, +storage, observability, and update-service providers may be processors or +recipients. Each deployer must identify the parties and roles for its +installation. + +References in this document are to the official text of the +[General Data Protection Regulation (GDPR), Regulation (EU) 2016/679](https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679). +The most relevant definitions are in Article 4. Pseudonymized identifiers remain +personal data when they can be attributed to a person using additional +information (Recital 26 and Articles 4(1) and 4(5)). + +## Scope and important limitations + +The inventory covers the application, its PostgreSQL schema, RabbitMQ email +messages, application logs, and optional OpenTelemetry metrics. It does not +define: + +- the identity and contact details of a deployment's controller, representative, + or data protection officer; +- the controller's purposes and lawful bases; +- all infrastructure logs created by a reverse proxy, firewall, load balancer, + container runtime, database, SMTP provider, object store, backup system, or + hosting provider; +- where a deployment or its subprocessors store data; +- whether data is transferred outside the European Economic Area; +- deployment-specific backup, log, and disaster-recovery retention periods; +- data that an operator enters into free-text fields or uploaded artifacts. + +Before using this as a public notice, the controller must add the information +required by Articles 13 or 14, including its identity, contact details, purposes, +lawful bases, recipients, transfers, retention criteria, rights, complaint +route, whether provision is required, and any automated decision-making. + +## Data subjects + +Clave processes or may receive data about: + +- license customers and users of licensed software; +- people who request self-service license access; +- administrators and organization members; +- people invited to an organization; +- people named or described in operator-entered metadata, release notes, + changelogs, filenames, URLs, logs, or uploaded files. + +Clave is not designed to collect special-category data under Article 9 or +criminal-conviction data under Article 10. An operator can nevertheless place +such data in free-text, JSON metadata, URLs, filenames, logs, or uploaded files. +Operators must not do so unless they have established an appropriate purpose, +lawful basis, and safeguards. + +## Data inventory + +### License customer data + +| Data | Source | Purpose in Clave | Storage and disclosure | Lifecycle | +| --- | --- | --- | --- | --- | +| Customer email address | Administrator or license workflow | Associate a license with a customer, send license and self-service emails, and locate licenses | Stored on the license; included in transactional email messages; masked in client activation responses | No automatic expiry; deleted through license, product, or organization deletion | +| Customer name | Administrator | Display the licensed customer's name | Stored on the license; returned by activation when present | No automatic expiry; follows license deletion | +| License key material | Generated by Clave | Authenticate activation and replacement workflows | Lookup HMAC/digest and Argon2 PHC hash are stored; plaintext keys can appear in one-time responses and transactional email queue messages | Hashes follow license deletion; queued license emails expire after seven days or are removed after successful delivery | +| License status and entitlements | Administrator and system | Enforce activation limits, validity, trials, and feature access | Product, expiry, active/trial state, activation limit, features, feature windows, and timestamps are stored | No automatic expiry; follows license, product, or organization deletion | +| Trial device digest | Client device | Enforce one trial per device and product | An HMAC-protected hardware identifier is stored on the trial license | No automatic expiry; follows license deletion | + +Clave does not automatically delete expired or revoked licenses. Keeping these +records supports licensing, abuse prevention, and support. The deployer must set +and document an appropriate retention rule under Article 5(1)(e). + +### Device, activation, and client synchronization data + +| Data | Source | Purpose in Clave | Storage and disclosure | Lifecycle | +| --- | --- | --- | --- | --- | +| Hardware identifier (`hwid`) | Licensed client | Recognize a device, enforce activation limits, and prevent repeat trials | The raw value is received by the API. Clave stores an HMAC-protected digest, not the raw value | Device records have no automatic expiry; they follow license deletion | +| Hostname | Licensed client | Identify a device for administrators, support, and self-service deactivation | Stored on the device and shown in device views | No automatic expiry; follows license deletion | +| Activation identifier and timestamps | Clave | Track activation lifecycle and authorize a device | Stores activation time, last-seen time, deactivation time, and optional reason | No automatic expiry; follows device or license deletion | +| Software environment | Licensed client | Validate and diagnose updates and show current version adoption | Version, build, platform, architecture, and OS version are stored as current activation state | No automatic expiry while the activation exists | +| Raw client check-in | Licensed client sync | Recent diagnostics and daily adoption aggregation | Links organization, product, license, and activation to version, build, platform, architecture, OS version, and time. It does not contain an IP address or User-Agent field | Default seven days; configurable from 1 to 30 days; deleted only after closed dates are aggregated | +| Daily version adoption | Derived by Clave | Long-term version trend reporting | Stores organization, product, UTC date, version, and device count; no activation, license, HWID, hostname, or customer identifier | No automatic expiry; follows organization or product deletion | + +A synchronization without a version still updates the activation's last-seen +time. A version-bearing synchronization updates current device state and creates +a raw check-in. The daily aggregate counts each activation once per UTC date +under its final reported version. + +Hardware digests, activation identifiers, hostnames, and software environments +can single out a device and can be linked to a license customer. Treat them as +personal data unless the controller can demonstrate that a natural person is not +identifiable by means reasonably likely to be used (Recital 26). IP addresses +and other online identifiers can also be personal data (Recital 30). + +### Update checks and staged rollout data + +| Data | Source | Purpose in Clave | Storage and disclosure | Lifecycle | +| --- | --- | --- | --- | --- | +| Update request environment | Licensed client | Select and diagnose an update | Platform, channel, current version, build, architecture, and OS version are stored with organization, product, and license references, provider, decision, selected release, and time | Default 90 days; configurable from 7 to 365 days | +| Stable rollout identifier (`clientId`) | Licensed client | Deterministically include or exclude a client from a percentage rollout | Used during request processing; the Clave `update_checks` table does not store it | Not intentionally persisted by Clave, but it can appear in infrastructure request logs if request bodies are logged | +| Artifact download request | Licensed client | Deliver an update | Clave authorizes local downloads; S3-backed storage redirects to a short-lived presigned URL | Clave has no dedicated completed-download table; proxy, object-store, or CDN logs can retain IP address, time, URL, and request metadata | + +The rollout decision is an automated decision based on the stable `clientId` and +a configured percentage. It controls update availability, including potentially +mandatory updates. The controller must assess whether Article 22 applies and, +regardless, explain material automated decision-making where Articles 13(2)(f) +or 14(2)(g) require it. + +### Self-service access data + +| Data | Source | Purpose in Clave | Storage and disclosure | Lifecycle | +| --- | --- | --- | --- | --- | +| Email address | Requester | Find licenses and send a passwordless access link | Stored with the temporary token; sent through RabbitMQ and the configured email provider | Token row is deleted one day after expiry or use; message expires after 15 minutes or is removed after delivery | +| Token and access metadata | Clave and HTTP request | Authenticate self-service access and investigate abuse | Stores a token hash, purpose, creation/use/expiry times, creation IP address, and optional User-Agent | Deleted one day after expiry or use | +| Self-service session cookie | Clave | Maintain authenticated self-service access | Browser receives `selfservice_session`; the cookie value contains the session credential used by the backend | One-hour browser lifetime; cleared on logout; it is not an advertising cookie | +| License-management actions | Requester | Revoke activations or replace a license | Changes activation/license state and can generate audit-relevant operational logs and an email containing a replacement key | Resulting license and activation records follow their normal lifecycle | + +### Administrator and organization data + +| Data | Source | Purpose in Clave | Storage and disclosure | Lifecycle | +| --- | --- | --- | --- | --- | +| Administrator account | Administrator | Authenticate and authorize administration | Email, Argon2 password hash, role, active state, last-login time, and account timestamps | No automatic expiry; deleted when the account is deleted | +| Organization membership | Administrator | Tenant isolation and role-based access | Organization name/slug, member identifiers, roles, and timestamps | Membership follows account or organization deletion | +| Organization invitation | Administrator and invitee | Invite a person and establish membership | Invitee email, role, hashed token, inviter, creation/expiry/acceptance times; email message contains the invitation link | Accepted invites are deleted after 30 days; unaccepted invites 30 days after expiry; queued email expires after seven days or delivery | +| Admin session | Clave | Maintain authenticated dashboard access | Browser receives `clave_admin_session`; PostgreSQL stores an opaque token, serialized session data, and expiry | 12-hour absolute lifetime, 30-minute idle timeout, logout deletion, and expired-row cleanup every five minutes | +| Admin email MFA code | Clave | Verify administrator login | Stores admin reference, HMAC code, attempts, creation/use/expiry times; plaintext code is sent through the email queue | Row deleted one day after expiry or use; queued message expires after 10 minutes or delivery | +| Legacy TOTP and recovery data | Administrator | Support deployments migrated from TOTP authentication | Schema can retain an encrypted TOTP secret and nonce, TOTP-enabled flag, and hashed recovery codes with creation/use times | No automatic expiry; follows administrator deletion | +| MCP credential metadata | Administrator and Clave | Authenticate organization-scoped MCP access | Hashed token, identifier, visible prefix, creator, regeneration time, and last-use time | Retained until regeneration or organization deletion | + +The dashboard uses necessary authentication, self-service, and CSRF controls. The +current codebase does not include advertising cookies or third-party behavioral +analytics scripts. A deployer can add such systems outside this repository and +must inventory them separately. + +### Audit, security, and application logs + +| Data | Source | Purpose in Clave | Storage and disclosure | Lifecycle | +| --- | --- | --- | --- | --- | +| Admin audit event | Administrator action and HTTP request | Accountability, security, and incident investigation | Actor, organization, action, resource type/identifier, free-form JSON metadata, timestamp, IP address, and User-Agent | IP and User-Agent default to 90 days, configurable from 30 to 180 days; core event defaults to 180 days, configurable from 90 to 365 days | +| Application log | Requests and runtime | Operations, debugging, and security | At debug level, request logs include method, path, remote address, User-Agent, status, duration, and request ID. Other logs can include email recipient, subject, internal identifiers, action names, and error text | Controlled by deployment log collection and rotation; Clave does not enforce log retention | +| Reverse-proxy and infrastructure log | HTTP and infrastructure | Delivery, operations, abuse prevention, and security | Can include IP address, timestamp, path, status, bytes, referrer, User-Agent, TLS details, and identifiers, depending on deployment | Controlled outside Clave | + +`TRUST_PROXY_HEADERS` allows Clave to use forwarded client addresses when it is +behind a trusted proxy. Enabling it behind an untrusted proxy can make recorded +addresses inaccurate. Debug logging increases personal-data exposure and must +not be enabled routinely in production. + +Free-form audit metadata and errors can contain more data than the predefined +schema. Operators must avoid placing secrets or unnecessary personal data in +these fields and logs. + +### Transactional email and queue data + +Clave can send: + +- administrator MFA codes; +- self-service magic links; +- organization invitations; +- created or replacement license keys. + +RabbitMQ messages can include recipient email, customer name, organization or +product name, link, code, or plaintext license key. Successful messages are +acknowledged and removed. Per-message expiration is 10 minutes for MFA, +15 minutes for self-service links, and seven days for invitations and license +emails. The configured SMTP or email provider receives the recipient, headers, +body, and delivery metadata and can maintain separate logs or message copies. +Its retention and international-transfer arrangements are deployment-specific. + +### Product, release, content, and storage data + +Administrators can store organization and product names, logos and URLs, +feature names and descriptions, update channels, provider/storage configuration, +release notes, changelogs, filenames, MIME types, artifact metadata, release +policies, and uploaded release artifacts. These are normally business or +technical data, but they become personal data if they identify or describe a +person. + +Provider and storage configuration is stored as JSON and can contain service +URLs, account identifiers, credentials, or other operator-supplied values. +Uploaded files and their metadata are not content-scanned for personal data. +Release artifacts can be stored locally or in a configured object store. + +Delta-generation jobs store release and artifact references, checksums, sizes, +status, error text, and timestamps. They follow the referenced release/artifact +lifecycle. Error text and artifact metadata are free-form and can contain +personal data if an operator includes it. + +### Metrics and observability + +When `OTEL_ENABLED=true`, Clave exports operational OpenTelemetry metrics to the +configured OTLP endpoint. Built-in metrics cover HTTP, database-pool, licensing, +audit, update-check, check-in, retention, and delta-job activity. The intended +metric attributes are operational categories and outcomes, not customer email, +hostname, HWID, IP address, or license key. + +Metric timestamps and low-volume label combinations can still reveal activity, +and an external collector can add resource attributes or correlate metrics with +logs. The observability provider, storage location, access controls, and +retention are deployment-specific. OpenTelemetry export is disabled by default. + +### Backups and replicas + +Database backups, object-store versions, snapshots, replicas, queue persistence, +and log archives can contain any data listed in this document. Deletion from the +live application does not necessarily remove data immediately from backups. +The controller must define backup retention, access, restoration behavior, and a +procedure that prevents erased data from being reintroduced permanently after a +restore. + +## Recipients and data flows + +Depending on configuration, personal data can be disclosed to: + +- the hosting, database, backup, and reverse-proxy providers; +- RabbitMQ infrastructure and the configured SMTP or transactional-email + provider; +- local or S3-compatible artifact storage, a CDN, and clients downloading + artifacts; +- update providers configured by an administrator; +- the configured OpenTelemetry collector and its backend; +- administrators and organization members authorized to use the dashboard; +- public authorities or other parties when required by applicable law. + +Articles 13(1)(e) and 14(1)(e) require the applicable notice to identify +recipients or categories of recipients. Article 28 governs processor contracts. +If data leaves the European Economic Area, Chapter V, including Articles 44 to 49, +requires an applicable transfer mechanism and safeguards. + +## Purposes and lawful bases + +Clave's code reveals functional purposes, but it cannot choose a controller's +lawful basis. The controller must map every processing purpose to an Article 6 +basis and document the analysis. Possible bases often assessed for a licensing +service include performance of a contract (Article 6(1)(b)), compliance with a +legal obligation (Article 6(1)(c)), and legitimate interests (Article 6(1)(f)). +This list is not a conclusion that any basis applies. + +If legitimate interests are used, document the purpose, necessity, balancing +test, safeguards, and right to object under Article 21. Do not use consent merely +because it is convenient; consent must meet Articles 4(11) and 7 and must be +freely given, specific, informed, unambiguous, and withdrawable. + +## GDPR principle cross-reference + +| GDPR requirement | Relevance to Clave | Deployment action | +| --- | --- | --- | +| Article 5(1)(a): lawfulness, fairness, transparency | Clave processes customer, device, admin, and network data | Publish an Article 13/14 notice and maintain a purpose-and-basis record | +| Article 5(1)(b): purpose limitation | Identifiers and diagnostics have defined licensing, update, security, and support purposes | Do not reuse them for unrelated analytics or profiling without a compatible purpose or new basis | +| Article 5(1)(c): data minimization | Raw HWID is transformed; check-ins omit IP/User-Agent; historical adoption is aggregated | Keep optional hostname and diagnostics only when needed; restrict free-form metadata and logs | +| Article 5(1)(d): accuracy | Hostname, customer data, and software state can become stale | Provide correction processes and distinguish historical records from current state | +| Article 5(1)(e): storage limitation | Some datasets have automatic retention, while licenses, devices, accounts, aggregates, content, logs, and backups do not | Set and enforce documented retention for every deployment-specific store | +| Articles 5(1)(f), 25, and 32: security and data protection by design/default | Clave hashes or HMAC-protects credentials and identifiers, supports TLS, and limits several retention windows | Protect secrets, enforce HTTPS, limit access, patch systems, test restores, and review logs and subprocessors | +| Article 5(2): accountability | Compliance depends on operator configuration and procedures | Keep records, decisions, contracts, tests, and evidence of deletion and access controls | +| Article 30: records of processing | The inventory can seed a record of processing activities | Add controller/processor roles, purposes, subjects, recipients, transfers, retention, and safeguards | +| Articles 33 and 34: breaches | Stored data and queued secrets can be exposed in an incident | Maintain detection, assessment, 72-hour authority-notification, and data-subject communication procedures | +| Article 35: impact assessment | Persistent device recognition and adoption monitoring can create elevated risk at scale | Assess whether a data protection impact assessment is required before deployment or material changes | + +## Data-subject rights + +Subject to exceptions and the controller's legal basis, GDPR rights include: + +- access (Article 15); +- rectification (Article 16); +- erasure (Article 17); +- restriction (Article 18); +- notification to recipients (Article 19); +- portability (Article 20); +- objection (Article 21); +- safeguards concerning qualifying automated decisions (Article 22); +- the right to complain to a supervisory authority (Article 77). + +Clave does not provide a complete automated data-subject request workflow. +Operators must verify identity, search linked records, account for pseudonymous +identifiers and backups, apply exceptions, notify recipients when required, and +respond within Article 12's deadlines. Relevant records can span customer +licenses, devices, activations, check-ins, update checks, self-service tokens, +admin accounts, memberships, invitations, audit logs, sessions, email queues, +application and proxy logs, metrics, provider logs, and backups. + +Erasing a license cascades to associated devices, activations, raw check-ins, +and feature grants in the live database. Erasing a product or organization has +broader cascading effects. Before deletion, assess contractual, security, +fraud-prevention, accounting, and legal-retention requirements. Aggregated daily +adoption has no stable device identifier, so it generally cannot be selected by +an individual device after aggregation; this does not by itself prove that the +aggregate is anonymous in every deployment. + +## Security measures visible in the codebase + +Clave includes these security measures: + +- HTTPS-only production guidance and secure, HTTP-only, SameSite cookies; +- Argon2 password and license-key verification hashes; +- HMAC or cryptographic hashes for hardware identifiers, temporary tokens, MFA + codes, recovery codes, and MCP credentials; +- encrypted legacy TOTP secrets; +- tenant-scoped authorization and audit logging; +- short-lived sessions, temporary credentials, and queued secret-bearing email; +- bounded retention for raw check-ins, update checks, audit data, invitations, + and temporary tokens; +- Ed25519-signed license tokens and update delta metadata. + +These measures do not establish GDPR compliance on their own. Article 32 +requires measures appropriate to the deployment's risks, including ongoing +confidentiality, integrity, availability, resilience, restoration, testing, +access control, key management, vulnerability management, and incident +response. + +## Operator checklist before publication + +1. Replace this technical inventory with, or incorporate it into, an Article 13 + and Article 14 notice written for the actual data subjects. +2. Name the controller, representative, data protection officer, and contact + route. +3. Record each purpose and Article 6 lawful basis; assess Articles 9, 10, and 22. +4. List processors and recipients, sign Article 28 agreements, and document any + Chapter V transfers. +5. Set retention for licenses, devices, administrator accounts, adoption + aggregates, content, logs, metrics, provider records, and backups. +6. Configure production retention variables and leave debug logging and + `SELF_SERVICE_RETURN_TOKEN` disabled. +7. Document data-subject request, deletion, backup, breach, and complaint + procedures. +8. Apply role-based access, TLS, secret management, encryption, monitoring, + patching, backup, and restore controls appropriate under Articles 25 and 32. +9. Review this inventory after schema, API, logging, provider, or infrastructure + changes. + +## Repository sources reviewed + +The review covered these sources: + +- `CLIENT.md`; +- `SETUP.md` and `.env.production.example`; +- `backend/migrations/`; +- `backend/internal/features/activation/`; +- `backend/internal/features/clientsync/`; +- `backend/internal/features/diagnostics/`; +- `backend/internal/features/selfservice/`; +- `backend/internal/features/adminauth/`; +- `backend/internal/features/audit/`; +- `backend/internal/features/update/`; +- `backend/internal/shared/events/`; +- `backend/internal/api/router.go` and `backend/internal/observability/`; +- `emailer/src/`. + +Last reviewed against repository commit `07d601f` and the official GDPR text +linked at the beginning of this document. diff --git a/SETUP.md b/SETUP.md index 3b8cb77..214d868 100644 --- a/SETUP.md +++ b/SETUP.md @@ -144,7 +144,10 @@ names is overridden. | `RUN_MIGRATIONS` | recommended | `false` | Run DB migrations at startup. Enable for exactly one instance per deployment. | | `PORT` | no | `8000` | API listen port inside the container. Changing it also requires changing the published port. | | `DATABASE_MAX_CONNS` | no | `20` | PostgreSQL pool size. Must be greater than zero. | -| `UPDATE_CHECK_RETENTION_DAYS` | no | `90` | Days of update-check history retained. `0` disables pruning. | +| `UPDATE_CHECK_RETENTION_DAYS` | no | `90` | Days of update-check history retained; must be 7–365. | +| `CLIENT_CHECKIN_RETENTION_DAYS` | no | `7` | Days of device-linked raw check-ins retained for diagnostics and aggregate retries; must be 1–30. Closed UTC dates are aggregated daily before eligible rows are deleted. | +| `AUDIT_LOG_RETENTION_DAYS` | no | `180` | Days core admin audit events remain identifiable; must be 90–365. | +| `AUDIT_METADATA_RETENTION_DAYS` | no | `90` | Days audit IP addresses and User-Agent strings are retained; must be 30–180 and no greater than audit-log retention. | | `SELF_SERVICE_RETURN_TOKEN` | no | `false` | Returns self-service tokens in API responses. Debug aid — leave off in production. | | `DEV` | no | unset | Development mode; relaxes key requirements. **Leave unset in production.** | | `LOG_LEVEL` | no | `info` | `debug`, `verbose`, or `trace` enable debug logging. | @@ -155,6 +158,48 @@ names is overridden. Truthy values are `true`, `1`, and `yes`. +### Client check-in data lifecycle + +An authorized client sync updates device-linked current state on the activation: +`last_seen_at`, version, build, platform, architecture, and OS version. Device +hostname and HMAC-protected HWID remain on the registered device for licensing, +support, and self-service deactivation. + +A sync that reports a version also creates a raw device-linked check-in without +IP address or User-Agent data. A daily worker aggregates every closed UTC date, normally +making yesterday's adoption counts available the next day. Each activation +counts once per date under its final reported version. Recomputing a date +replaces its aggregate, so retries cannot double-count devices. + +Raw check-ins remain for `CLIENT_CHECKIN_RETENTION_DAYS` as a diagnostic and +retry window, then are deleted only after aggregation succeeds. Long-term +`daily_version_adoption` rows retain organization, product, date, version, and +count, but no activation, license, HWID, hostname, or customer identifier. + +### Persisted data retention + +A centralized worker runs at startup and daily. Cleanup statements are +idempotent, failures are isolated per dataset, and affected row counts are +logged. OpenTelemetry records cleanup outcomes by dataset. Cleanup never runs +on request paths. + +| Data | Purpose and sensitivity | Lifecycle | +| --- | --- | --- | +| Self-service tokens | Email, hashed token, expiry, consumption time, creation IP, and optional User-Agent for passwordless access and abuse investigation. | Delete one day after expiry or consumption. | +| Admin sessions | Opaque token and serialized admin session used for authenticated access. | Absolute lifetime 12 hours, idle timeout 30 minutes, immediate deletion on logout, and pgxstore cleanup of expired rows every five minutes. | +| Organization invites | Invitee email, role, hashed token, inviter, expiry, and acceptance time. | Delete accepted invites after 30 days; delete unaccepted invites 30 days after expiry. | +| Admin MFA codes | Admin reference, HMAC code, attempts, expiry, and use time. | Delete one day after expiry or consumption. | +| Admin audit events | Actor, organization, action, resource, timestamp, IP, and User-Agent for accountability and incident investigation. | Scrub IP and User-Agent after `AUDIT_METADATA_RETENTION_DAYS`; delete the core event after `AUDIT_LOG_RETENTION_DAYS`. | +| Update checks | Organization/product/license references and software environment used for update diagnostics. | Delete after `UPDATE_CHECK_RETENTION_DAYS`. | +| Email queue messages | Recipient and transactional payload, which may include MFA codes, links, or license keys. | RabbitMQ removes successful messages immediately. Per-message expiry is 10 minutes for MFA, 15 minutes for self-service links, and seven days for organization invites and license emails. No completed-delivery table is kept. | +| MCP tokens | Hashed organization credential plus creator and last-use metadata. | Retain until regeneration or organization deletion. | +| Delta jobs | Release/artifact checksums and operational status used for idempotency and diagnostics. | Retain with the referenced release/artifact lifecycle. | + +Expired and revoked licenses, their devices, and admin accounts are deliberately +excluded from automated retention. Their identity supports licensing, abuse +prevention, customer support, and multi-organization access. They require a +separate product/legal decision before deletion or de-identification. + Ensure `RUN_MIGRATIONS=true` for exactly one backend instance during a deployment, then start it: diff --git a/backend/internal/app/app.go b/backend/internal/app/app.go index 8c75ada..d68134f 100644 --- a/backend/internal/app/app.go +++ b/backend/internal/app/app.go @@ -25,6 +25,7 @@ import ( "github.com/cheetahbyte/clave/internal/features/update/providers/native" "github.com/cheetahbyte/clave/internal/features/validation" "github.com/cheetahbyte/clave/internal/observability" + "github.com/cheetahbyte/clave/internal/retention" "github.com/cheetahbyte/clave/internal/shared/events" "github.com/cheetahbyte/clave/internal/shared/helpers" "github.com/cheetahbyte/clave/internal/shared/middleware" @@ -42,6 +43,7 @@ var ( publisher *events.Publisher updateRecorder *update.UpdateCheckRecorder checkinRecorder *diagnostics.Recorder + retentionWorker *retention.Worker ) func Close() { @@ -53,6 +55,9 @@ func Close() { if checkinRecorder != nil { checkinRecorder.Close(ctx) } + if retentionWorker != nil { + retentionWorker.Close(ctx) + } observability.Shutdown(ctx) if publisher != nil { publisher.Close() @@ -105,6 +110,11 @@ func NewRouter(cfg *config.Config) (http.Handler, error) { observability.StartDBPoolMetrics(context.Background(), pool, 30*time.Second) q := db.New(pool) + retentionWorker = retention.NewWorker(q, retention.Policies{ + AuditMetadataDays: cfg.AuditMetadataRetentionDays, + AuditLogDays: cfg.AuditLogRetentionDays, + UpdateCheckDays: cfg.UpdateCheckRetentionDays, + }) signer := signing.New(cfg.LicenseJWTPublicKey, cfg.LicenseJWTPrivateKey, cfg.LicenseHMACSecret) @@ -117,14 +127,14 @@ func NewRouter(cfg *config.Config) (http.Handler, error) { ) updateSvc := update.NewService(licenseSvc, signer, updateRepo, updateRegistry, cfg.PublicAppURL, cfg.UpdateArtifactStoragePath) - updateRecorder = update.NewUpdateCheckRecorder(updateRepo, cfg.UpdateCheckRetentionDays, 256) + updateRecorder = update.NewUpdateCheckRecorder(updateRepo, 256) updateSvc.SetCheckRecorder(updateRecorder) validationSvc := validation.NewService(q, signer, licenseSvc, updateSvc) activationSvc := activation.NewService(q, pool, signer, licenseSvc, updateSvc, validationSvc) updateSvc.SetValidator(validationSvc) - diagnosticsRepo := diagnostics.NewRepository(q) + diagnosticsRepo := diagnostics.NewRepository(q, pool) diagnosticsSvc := diagnostics.NewService(diagnosticsRepo) - checkinRecorder = diagnostics.NewRecorder(diagnosticsRepo, 90, 256) + checkinRecorder = diagnostics.NewRecorder(diagnosticsRepo, cfg.ClientCheckinRetentionDays) clientSyncSvc := clientsync.NewService(validationSvc, updateSvc, checkinRecorder) selfServiceRepo := selfservice.NewRepository(q, pool) selfserviceSvc := selfservice.NewService(selfServiceRepo, []byte(cfg.SelfServiceTokenPepper), signer, licenseSvc, updateSvc) diff --git a/backend/internal/config/config.go b/backend/internal/config/config.go index d46d85e..ce548e9 100644 --- a/backend/internal/config/config.go +++ b/backend/internal/config/config.go @@ -9,6 +9,7 @@ import ( "errors" "fmt" "log/slog" + "math" "os" "strconv" "strings" @@ -49,8 +50,11 @@ type Config struct { TrustProxyHeaders bool - UpdateArtifactStoragePath string - UpdateCheckRetentionDays int + UpdateArtifactStoragePath string + UpdateCheckRetentionDays int + ClientCheckinRetentionDays int + AuditLogRetentionDays int + AuditMetadataRetentionDays int MigrationsDir string OTELEnabled bool @@ -82,43 +86,76 @@ func getEnvInt(key string, fallback int) (int, error) { return value, nil } +func getEnvIntRange(key string, fallback, min, max int) (int, error) { + value, err := getEnvInt(key, fallback) + if err != nil { + return 0, err + } + if value < min || value > max { + return 0, fmt.Errorf("%s must be between %d and %d", key, min, max) + } + return value, nil +} + func Load() (*Config, error) { databaseMaxConns, err := getEnvInt("DATABASE_MAX_CONNS", 20) - if err != nil || databaseMaxConns == 0 { + if err != nil || databaseMaxConns == 0 || databaseMaxConns > math.MaxInt32 { if err == nil { - err = fmt.Errorf("DATABASE_MAX_CONNS must be greater than zero") + if databaseMaxConns == 0 { + err = fmt.Errorf("DATABASE_MAX_CONNS must be greater than zero") + } else { + err = fmt.Errorf("DATABASE_MAX_CONNS must be less than or equal to %d", math.MaxInt32) + } } return nil, err } - retentionDays, err := getEnvInt("UPDATE_CHECK_RETENTION_DAYS", 90) + retentionDays, err := getEnvIntRange("UPDATE_CHECK_RETENTION_DAYS", 90, 7, 365) + if err != nil { + return nil, err + } + checkinRetentionDays, err := getEnvIntRange("CLIENT_CHECKIN_RETENTION_DAYS", 7, 1, 30) + if err != nil { + return nil, err + } + auditLogRetentionDays, err := getEnvIntRange("AUDIT_LOG_RETENTION_DAYS", 180, 90, 365) if err != nil { return nil, err } + auditMetadataRetentionDays, err := getEnvIntRange("AUDIT_METADATA_RETENTION_DAYS", 90, 30, 180) + if err != nil { + return nil, err + } + if auditMetadataRetentionDays > auditLogRetentionDays { + return nil, errors.New("AUDIT_METADATA_RETENTION_DAYS must not exceed AUDIT_LOG_RETENTION_DAYS") + } cfg := &Config{ - DatabaseURL: getEnv("DATABASE_URL", "postgres://clave@localhost:54321/clave?sslmode=disable"), - DatabaseMaxConns: int32(databaseMaxConns), - RunMigrations: truthy(os.Getenv("RUN_MIGRATIONS")), - VerboseLogging: verboseLoggingEnabled(), - Dev: truthy(os.Getenv("DEV")), - MigrationsDir: getEnv("MIGRATIONS_DIR", "./migrations"), - LicenseHMACSecret: os.Getenv("LICENSE_HMAC_SECRET"), - OTELEnabled: truthy(os.Getenv("OTEL_ENABLED")), - OTELServiceName: getEnv("OTEL_SERVICE_NAME", "clave-api"), - OTELExporterEndpoint: os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"), - SelfServiceTokenPepper: os.Getenv("SELF_SERVICE_TOKEN_PEPPER"), - SelfServiceReturnToken: strings.ToLower(os.Getenv("SELF_SERVICE_RETURN_TOKEN")) == "true", - SMTPHost: os.Getenv("SMTP_HOST"), - SMTPPort: getEnv("SMTP_PORT", "587"), - SMTPUser: os.Getenv("SMTP_USER"), - SMTPPass: os.Getenv("SMTP_PASS"), - MailFrom: getEnv("MAIL_FROM", "noreply@clave.app"), - RabbitMQURL: os.Getenv("RABBITMQ_URL"), - WorkerToken: os.Getenv("WORKER_TOKEN"), - PublicAppURL: os.Getenv("PUBLIC_APP_URL"), - Port: getEnv("PORT", "8000"), - TrustProxyHeaders: truthy(os.Getenv("TRUST_PROXY_HEADERS")), - UpdateArtifactStoragePath: getEnv("UPDATE_ARTIFACT_STORAGE_PATH", "./data/update-artifacts"), - UpdateCheckRetentionDays: retentionDays, + DatabaseURL: getEnv("DATABASE_URL", "postgres://clave@localhost:54321/clave?sslmode=disable"), + DatabaseMaxConns: int32(databaseMaxConns), + RunMigrations: truthy(os.Getenv("RUN_MIGRATIONS")), + VerboseLogging: verboseLoggingEnabled(), + Dev: truthy(os.Getenv("DEV")), + MigrationsDir: getEnv("MIGRATIONS_DIR", "./migrations"), + LicenseHMACSecret: os.Getenv("LICENSE_HMAC_SECRET"), + OTELEnabled: truthy(os.Getenv("OTEL_ENABLED")), + OTELServiceName: getEnv("OTEL_SERVICE_NAME", "clave-api"), + OTELExporterEndpoint: os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"), + SelfServiceTokenPepper: os.Getenv("SELF_SERVICE_TOKEN_PEPPER"), + SelfServiceReturnToken: strings.ToLower(os.Getenv("SELF_SERVICE_RETURN_TOKEN")) == "true", + SMTPHost: os.Getenv("SMTP_HOST"), + SMTPPort: getEnv("SMTP_PORT", "587"), + SMTPUser: os.Getenv("SMTP_USER"), + SMTPPass: os.Getenv("SMTP_PASS"), + MailFrom: getEnv("MAIL_FROM", "noreply@clave.app"), + RabbitMQURL: os.Getenv("RABBITMQ_URL"), + WorkerToken: os.Getenv("WORKER_TOKEN"), + PublicAppURL: os.Getenv("PUBLIC_APP_URL"), + Port: getEnv("PORT", "8000"), + TrustProxyHeaders: truthy(os.Getenv("TRUST_PROXY_HEADERS")), + UpdateArtifactStoragePath: getEnv("UPDATE_ARTIFACT_STORAGE_PATH", "./data/update-artifacts"), + UpdateCheckRetentionDays: retentionDays, + ClientCheckinRetentionDays: checkinRetentionDays, + AuditLogRetentionDays: auditLogRetentionDays, + AuditMetadataRetentionDays: auditMetadataRetentionDays, } cfg.DevSkip2FA = cfg.Dev && !truthy(os.Getenv("DEV_FORCE_2FA")) diff --git a/backend/internal/config/config_test.go b/backend/internal/config/config_test.go index 4e31619..f13fa6b 100644 --- a/backend/internal/config/config_test.go +++ b/backend/internal/config/config_test.go @@ -46,6 +46,19 @@ func TestGetEnvInt(t *testing.T) { } } +func TestGetEnvIntRange(t *testing.T) { + t.Setenv("CLAVE_TEST_RANGE", "") + if got, err := getEnvIntRange("CLAVE_TEST_RANGE", 90, 7, 365); err != nil || got != 90 { + t.Fatalf("default = %d, %v", got, err) + } + for _, value := range []string{"0", "366", "invalid"} { + t.Setenv("CLAVE_TEST_RANGE", value) + if _, err := getEnvIntRange("CLAVE_TEST_RANGE", 90, 7, 365); err == nil { + t.Fatalf("expected %q to fail", value) + } + } +} + func TestLoadEd25519PrivateKeyFileErrors(t *testing.T) { rsaKey, err := rsa.GenerateKey(rand.Reader, 2048) if err != nil { diff --git a/backend/internal/db/activations.sql.go b/backend/internal/db/activations.sql.go index ece1c73..37aa6a0 100644 --- a/backend/internal/db/activations.sql.go +++ b/backend/internal/db/activations.sql.go @@ -13,7 +13,7 @@ import ( ) const activateLicense = `-- name: ActivateLicense :one -insert into activations (device_id, license_id) values($1, $2) returning device_id, checked_in_at, created_at, id, license_id, deactivated_at, deactivation_reason +insert into activations (device_id, license_id) values($1, $2) returning device_id, checked_in_at, created_at, id, license_id, deactivated_at, deactivation_reason, last_seen_at, current_version, current_build, platform, arch, os_version ` type ActivateLicenseParams struct { @@ -32,6 +32,12 @@ func (q *Queries) ActivateLicense(ctx context.Context, arg ActivateLicenseParams &i.LicenseID, &i.DeactivatedAt, &i.DeactivationReason, + &i.LastSeenAt, + &i.CurrentVersion, + &i.CurrentBuild, + &i.Platform, + &i.Arch, + &i.OsVersion, ) return i, err } @@ -73,7 +79,7 @@ func (q *Queries) DeactivateActivationByLicenseAndHwid(ctx context.Context, arg } const getActivationByLicenseAndDevice = `-- name: GetActivationByLicenseAndDevice :one -select device_id, checked_in_at, created_at, id, license_id, deactivated_at, deactivation_reason from activations where license_id = $1 and device_id = $2 and deactivated_at is null +select device_id, checked_in_at, created_at, id, license_id, deactivated_at, deactivation_reason, last_seen_at, current_version, current_build, platform, arch, os_version from activations where license_id = $1 and device_id = $2 and deactivated_at is null ` type GetActivationByLicenseAndDeviceParams struct { @@ -92,12 +98,18 @@ func (q *Queries) GetActivationByLicenseAndDevice(ctx context.Context, arg GetAc &i.LicenseID, &i.DeactivatedAt, &i.DeactivationReason, + &i.LastSeenAt, + &i.CurrentVersion, + &i.CurrentBuild, + &i.Platform, + &i.Arch, + &i.OsVersion, ) return i, err } const getActivationsForLicense = `-- name: GetActivationsForLicense :many -select device_id, checked_in_at, created_at, id, license_id, deactivated_at, deactivation_reason from activations where license_id = $1 and deactivated_at is null +select device_id, checked_in_at, created_at, id, license_id, deactivated_at, deactivation_reason, last_seen_at, current_version, current_build, platform, arch, os_version from activations where license_id = $1 and deactivated_at is null ` func (q *Queries) GetActivationsForLicense(ctx context.Context, licenseID uuid.UUID) ([]Activation, error) { @@ -117,6 +129,12 @@ func (q *Queries) GetActivationsForLicense(ctx context.Context, licenseID uuid.U &i.LicenseID, &i.DeactivatedAt, &i.DeactivationReason, + &i.LastSeenAt, + &i.CurrentVersion, + &i.CurrentBuild, + &i.Platform, + &i.Arch, + &i.OsVersion, ); err != nil { return nil, err } @@ -129,7 +147,7 @@ func (q *Queries) GetActivationsForLicense(ctx context.Context, licenseID uuid.U } const getActiveActivationByID = `-- name: GetActiveActivationByID :one -select a.device_id, a.checked_in_at, a.created_at, a.id, a.license_id, a.deactivated_at, a.deactivation_reason +select a.device_id, a.checked_in_at, a.created_at, a.id, a.license_id, a.deactivated_at, a.deactivation_reason, a.last_seen_at, a.current_version, a.current_build, a.platform, a.arch, a.os_version from activations a join devices d on d.id = a.device_id where a.id = $1 @@ -155,12 +173,18 @@ func (q *Queries) GetActiveActivationByID(ctx context.Context, arg GetActiveActi &i.LicenseID, &i.DeactivatedAt, &i.DeactivationReason, + &i.LastSeenAt, + &i.CurrentVersion, + &i.CurrentBuild, + &i.Platform, + &i.Arch, + &i.OsVersion, ) return i, err } const getActiveActivationByLicenseAndHwidHash = `-- name: GetActiveActivationByLicenseAndHwidHash :one -select a.device_id, a.checked_in_at, a.created_at, a.id, a.license_id, a.deactivated_at, a.deactivation_reason +select a.device_id, a.checked_in_at, a.created_at, a.id, a.license_id, a.deactivated_at, a.deactivation_reason, a.last_seen_at, a.current_version, a.current_build, a.platform, a.arch, a.os_version from activations a join devices d on d.id = a.device_id where a.license_id = $1 @@ -184,6 +208,12 @@ func (q *Queries) GetActiveActivationByLicenseAndHwidHash(ctx context.Context, a &i.LicenseID, &i.DeactivatedAt, &i.DeactivationReason, + &i.LastSeenAt, + &i.CurrentVersion, + &i.CurrentBuild, + &i.Platform, + &i.Arch, + &i.OsVersion, ) return i, err } diff --git a/backend/internal/db/admin.sql.go b/backend/internal/db/admin.sql.go index 3591650..35d5762 100644 --- a/backend/internal/db/admin.sql.go +++ b/backend/internal/db/admin.sql.go @@ -78,14 +78,75 @@ func (q *Queries) CreateOrganizationMember(ctx context.Context, arg CreateOrgani return err } -const deleteExpiredAdminEmailCodes = `-- name: DeleteExpiredAdminEmailCodes :exec +const deleteStaleAdminEmailCodes = `-- name: DeleteStaleAdminEmailCodes :execrows DELETE FROM admin_email_codes WHERE expires_at < now() - INTERVAL '1 day' + OR used_at < now() - INTERVAL '1 day' ` -func (q *Queries) DeleteExpiredAdminEmailCodes(ctx context.Context) error { - _, err := q.db.Exec(ctx, deleteExpiredAdminEmailCodes) - return err +func (q *Queries) DeleteStaleAdminEmailCodes(ctx context.Context) (int64, error) { + result, err := q.db.Exec(ctx, deleteStaleAdminEmailCodes) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const deleteStaleAuditLogs = `-- name: DeleteStaleAuditLogs :execrows +DELETE FROM admin_audit_log +WHERE created_at < now() - make_interval(days => $1::int) +` + +func (q *Queries) DeleteStaleAuditLogs(ctx context.Context, retentionDays int32) (int64, error) { + result, err := q.db.Exec(ctx, deleteStaleAuditLogs, retentionDays) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const deleteStaleOrganizationInvites = `-- name: DeleteStaleOrganizationInvites :execrows +DELETE FROM organization_invites +WHERE (accepted_at IS NOT NULL AND accepted_at < now() - INTERVAL '30 days') + OR (accepted_at IS NULL AND expires_at < now() - INTERVAL '30 days') +` + +func (q *Queries) DeleteStaleOrganizationInvites(ctx context.Context) (int64, error) { + result, err := q.db.Exec(ctx, deleteStaleOrganizationInvites) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const deleteStaleSelfServiceTokens = `-- name: DeleteStaleSelfServiceTokens :execrows + +DELETE FROM self_service_tokens +WHERE expires_at < now() - INTERVAL '1 day' + OR used_at < now() - INTERVAL '1 day' +` + +// Retention maintenance queries are centralized here so one generated query +// adapter can run the database lifecycle worker. +func (q *Queries) DeleteStaleSelfServiceTokens(ctx context.Context) (int64, error) { + result, err := q.db.Exec(ctx, deleteStaleSelfServiceTokens) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const deleteStaleUpdateChecks = `-- name: DeleteStaleUpdateChecks :execrows +DELETE FROM update_checks +WHERE created_at < now() - make_interval(days => $1::int) +` + +func (q *Queries) DeleteStaleUpdateChecks(ctx context.Context, retentionDays int32) (int64, error) { + result, err := q.db.Exec(ctx, deleteStaleUpdateChecks, retentionDays) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil } const getAdminByEmail = `-- name: GetAdminByEmail :one @@ -334,6 +395,22 @@ func (q *Queries) MarkAdminEmailCodeUsed(ctx context.Context, id uuid.UUID) erro return err } +const scrubStaleAuditSecurityMetadata = `-- name: ScrubStaleAuditSecurityMetadata :execrows +UPDATE admin_audit_log +SET ip = NULL, + user_agent = NULL +WHERE created_at < now() - make_interval(days => $1::int) + AND (ip IS NOT NULL OR user_agent IS NOT NULL) +` + +func (q *Queries) ScrubStaleAuditSecurityMetadata(ctx context.Context, retentionDays int32) (int64, error) { + result, err := q.db.Exec(ctx, scrubStaleAuditSecurityMetadata, retentionDays) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + const updateLastLogin = `-- name: UpdateLastLogin :exec UPDATE admin_users SET last_login_at = now(), updated_at = now() diff --git a/backend/internal/db/diagnostics.sql.go b/backend/internal/db/diagnostics.sql.go index 44a29fb..e0cbac1 100644 --- a/backend/internal/db/diagnostics.sql.go +++ b/backend/internal/db/diagnostics.sql.go @@ -12,6 +12,16 @@ import ( "github.com/jackc/pgx/v5/pgtype" ) +const deleteDailyVersionAdoptionForDate = `-- name: DeleteDailyVersionAdoptionForDate :exec +DELETE FROM daily_version_adoption +WHERE date = $1::date +` + +func (q *Queries) DeleteDailyVersionAdoptionForDate(ctx context.Context, date pgtype.Date) error { + _, err := q.db.Exec(ctx, deleteDailyVersionAdoptionForDate, date) + return err +} + const deleteExpiredClientCheckins = `-- name: DeleteExpiredClientCheckins :execrows DELETE FROM client_checkins WHERE created_at < now() - make_interval(days => $1::int) @@ -25,103 +35,48 @@ func (q *Queries) DeleteExpiredClientCheckins(ctx context.Context, retentionDays return result.RowsAffected(), nil } -const insertClientCheckin = `-- name: InsertClientCheckin :one -INSERT INTO client_checkins ( - organization_id, product_id, license_id, activation_id, - version, build, platform, arch, os_version -) -VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) -RETURNING id, organization_id, product_id, license_id, activation_id, version, build, platform, arch, os_version, created_at -` - -type InsertClientCheckinParams struct { - OrganizationID uuid.UUID `json:"organization_id"` - ProductID uuid.UUID `json:"product_id"` - LicenseID uuid.UUID `json:"license_id"` - ActivationID uuid.UUID `json:"activation_id"` - Version string `json:"version"` - Build *string `json:"build"` - Platform *string `json:"platform"` - Arch *string `json:"arch"` - OsVersion *string `json:"os_version"` -} - -func (q *Queries) InsertClientCheckin(ctx context.Context, arg InsertClientCheckinParams) (ClientCheckin, error) { - row := q.db.QueryRow(ctx, insertClientCheckin, - arg.OrganizationID, - arg.ProductID, - arg.LicenseID, - arg.ActivationID, - arg.Version, - arg.Build, - arg.Platform, - arg.Arch, - arg.OsVersion, - ) - var i ClientCheckin - err := row.Scan( - &i.ID, - &i.OrganizationID, - &i.ProductID, - &i.LicenseID, - &i.ActivationID, - &i.Version, - &i.Build, - &i.Platform, - &i.Arch, - &i.OsVersion, - &i.CreatedAt, - ) - return i, err -} - -const listDailyLatestClientVersions = `-- name: ListDailyLatestClientVersions :many -SELECT date, version, count(*)::bigint AS device_count +const insertDailyVersionAdoptionForDate = `-- name: InsertDailyVersionAdoptionForDate :exec +INSERT INTO daily_version_adoption (date, organization_id, product_id, version, device_count) +SELECT observed_date, organization_id, product_id, version, count(*) FROM ( - SELECT DISTINCT ON (c.created_at::date, c.activation_id) - c.created_at::date AS date, - c.activation_id, - c.version - FROM client_checkins c - JOIN activations a ON a.id = c.activation_id - WHERE c.organization_id = $1::uuid - AND c.created_at >= now() - make_interval(days => $2::int) - AND ( - $3::uuid IS NULL - OR c.product_id = $3::uuid - ) - AND a.deactivated_at IS NULL - ORDER BY c.created_at::date, c.activation_id, c.created_at DESC, c.id DESC -) latest -GROUP BY date, version -ORDER BY date, version + SELECT DISTINCT ON (activation_id) + (created_at AT TIME ZONE 'UTC')::date AS observed_date, + organization_id, + product_id, + activation_id, + version + FROM client_checkins + WHERE (created_at AT TIME ZONE 'UTC')::date = $1::date + ORDER BY activation_id, created_at DESC, id DESC +) latest_daily +GROUP BY observed_date, organization_id, product_id, version ` -type ListDailyLatestClientVersionsParams struct { - OrganizationID uuid.UUID `json:"organization_id"` - Days int32 `json:"days"` - ProductID pgtype.UUID `json:"product_id"` +func (q *Queries) InsertDailyVersionAdoptionForDate(ctx context.Context, date pgtype.Date) error { + _, err := q.db.Exec(ctx, insertDailyVersionAdoptionForDate, date) + return err } -type ListDailyLatestClientVersionsRow struct { - Date pgtype.Date `json:"date"` - Version string `json:"version"` - DeviceCount int64 `json:"device_count"` -} +const listClosedClientCheckinDates = `-- name: ListClosedClientCheckinDates :many +SELECT DISTINCT (created_at AT TIME ZONE 'UTC')::date AS date +FROM client_checkins +WHERE (created_at AT TIME ZONE 'UTC')::date < (now() AT TIME ZONE 'UTC')::date +ORDER BY date +` -func (q *Queries) ListDailyLatestClientVersions(ctx context.Context, arg ListDailyLatestClientVersionsParams) ([]ListDailyLatestClientVersionsRow, error) { - rows, err := q.db.Query(ctx, listDailyLatestClientVersions, arg.OrganizationID, arg.Days, arg.ProductID) +func (q *Queries) ListClosedClientCheckinDates(ctx context.Context) ([]pgtype.Date, error) { + rows, err := q.db.Query(ctx, listClosedClientCheckinDates) if err != nil { return nil, err } defer rows.Close() - items := []ListDailyLatestClientVersionsRow{} + items := []pgtype.Date{} for rows.Next() { - var i ListDailyLatestClientVersionsRow - if err := rows.Scan(&i.Date, &i.Version, &i.DeviceCount); err != nil { + var date pgtype.Date + if err := rows.Scan(&date); err != nil { return nil, err } - items = append(items, i) + items = append(items, date) } if err := rows.Err(); err != nil { return nil, err @@ -129,65 +84,56 @@ func (q *Queries) ListDailyLatestClientVersions(ctx context.Context, arg ListDai return items, nil } -const listLatestClientCheckins = `-- name: ListLatestClientCheckins :many -WITH ranked AS ( - SELECT - c.id, c.organization_id, c.product_id, c.license_id, c.activation_id, c.version, c.build, c.platform, c.arch, c.os_version, c.created_at, - row_number() OVER ( - PARTITION BY c.activation_id - ORDER BY c.created_at DESC, c.id DESC - ) AS position - FROM client_checkins c - WHERE c.organization_id = $1::uuid - AND c.created_at >= now() - make_interval(days => $2::int) - AND ( - $3::uuid IS NULL - OR c.product_id = $3::uuid - ) -) +const listCurrentClientStates = `-- name: ListCurrentClientStates :many SELECT - ranked.activation_id, + a.id AS activation_id, d.hostname, - ranked.version, - ranked.build, - ranked.platform, - ranked.arch, - ranked.os_version, - ranked.created_at -FROM ranked -JOIN activations a ON a.id = ranked.activation_id + a.current_version AS version, + a.current_build AS build, + a.platform, + a.arch, + a.os_version, + a.last_seen_at +FROM activations a JOIN devices d ON d.id = a.device_id -WHERE ranked.position = 1 +JOIN licenses l ON l.id = a.license_id +WHERE l.organization_id = $1::uuid AND a.deactivated_at IS NULL -ORDER BY ranked.created_at DESC + AND a.last_seen_at >= now() - make_interval(days => $2::int) + AND a.current_version IS NOT NULL + AND ( + $3::uuid IS NULL + OR l.product_id = $3::uuid + ) +ORDER BY a.last_seen_at DESC, a.id ` -type ListLatestClientCheckinsParams struct { +type ListCurrentClientStatesParams struct { OrganizationID uuid.UUID `json:"organization_id"` Days int32 `json:"days"` ProductID pgtype.UUID `json:"product_id"` } -type ListLatestClientCheckinsRow struct { +type ListCurrentClientStatesRow struct { ActivationID uuid.UUID `json:"activation_id"` Hostname *string `json:"hostname"` - Version string `json:"version"` + Version *string `json:"version"` Build *string `json:"build"` Platform *string `json:"platform"` Arch *string `json:"arch"` OsVersion *string `json:"os_version"` - CreatedAt pgtype.Timestamptz `json:"created_at"` + LastSeenAt pgtype.Timestamptz `json:"last_seen_at"` } -func (q *Queries) ListLatestClientCheckins(ctx context.Context, arg ListLatestClientCheckinsParams) ([]ListLatestClientCheckinsRow, error) { - rows, err := q.db.Query(ctx, listLatestClientCheckins, arg.OrganizationID, arg.Days, arg.ProductID) +func (q *Queries) ListCurrentClientStates(ctx context.Context, arg ListCurrentClientStatesParams) ([]ListCurrentClientStatesRow, error) { + rows, err := q.db.Query(ctx, listCurrentClientStates, arg.OrganizationID, arg.Days, arg.ProductID) if err != nil { return nil, err } defer rows.Close() - items := []ListLatestClientCheckinsRow{} + items := []ListCurrentClientStatesRow{} for rows.Next() { - var i ListLatestClientCheckinsRow + var i ListCurrentClientStatesRow if err := rows.Scan( &i.ActivationID, &i.Hostname, @@ -196,7 +142,7 @@ func (q *Queries) ListLatestClientCheckins(ctx context.Context, arg ListLatestCl &i.Platform, &i.Arch, &i.OsVersion, - &i.CreatedAt, + &i.LastSeenAt, ); err != nil { return nil, err } @@ -207,3 +153,101 @@ func (q *Queries) ListLatestClientCheckins(ctx context.Context, arg ListLatestCl } return items, nil } + +const listDailyVersionAdoption = `-- name: ListDailyVersionAdoption :many +SELECT date, version, sum(device_count)::bigint AS device_count +FROM daily_version_adoption +WHERE organization_id = $1::uuid + AND date >= ((now() AT TIME ZONE 'UTC')::date - ($2::int - 1)) + AND ( + $3::uuid IS NULL + OR product_id = $3::uuid + ) +GROUP BY date, version +ORDER BY date, version +` + +type ListDailyVersionAdoptionParams struct { + OrganizationID uuid.UUID `json:"organization_id"` + Days int32 `json:"days"` + ProductID pgtype.UUID `json:"product_id"` +} + +type ListDailyVersionAdoptionRow struct { + Date pgtype.Date `json:"date"` + Version string `json:"version"` + DeviceCount int64 `json:"device_count"` +} + +func (q *Queries) ListDailyVersionAdoption(ctx context.Context, arg ListDailyVersionAdoptionParams) ([]ListDailyVersionAdoptionRow, error) { + rows, err := q.db.Query(ctx, listDailyVersionAdoption, arg.OrganizationID, arg.Days, arg.ProductID) + if err != nil { + return nil, err + } + defer rows.Close() + items := []ListDailyVersionAdoptionRow{} + for rows.Next() { + var i ListDailyVersionAdoptionRow + if err := rows.Scan(&i.Date, &i.Version, &i.DeviceCount); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const recordClientCheckin = `-- name: RecordClientCheckin :exec +WITH updated AS ( + UPDATE activations a + SET last_seen_at = now(), + current_version = CASE WHEN $4 <> '' THEN $4 ELSE a.current_version END, + current_build = CASE WHEN $4 <> '' THEN $5 ELSE a.current_build END, + platform = CASE WHEN $4 <> '' THEN $6 ELSE a.platform END, + arch = CASE WHEN $4 <> '' THEN $7 ELSE a.arch END, + os_version = CASE WHEN $4 <> '' THEN $8 ELSE a.os_version END + WHERE a.id = $9 + AND a.license_id = $3 + AND a.deactivated_at IS NULL + RETURNING a.id +) +INSERT INTO client_checkins ( + organization_id, product_id, license_id, activation_id, + version, build, platform, arch, os_version +) +SELECT + $1, $2, $3, updated.id, + $4, $5, $6, + $7, $8 +FROM updated +WHERE $4 <> '' +` + +type RecordClientCheckinParams struct { + OrganizationID uuid.UUID `json:"organization_id"` + ProductID uuid.UUID `json:"product_id"` + LicenseID uuid.UUID `json:"license_id"` + Version string `json:"version"` + Build *string `json:"build"` + Platform *string `json:"platform"` + Arch *string `json:"arch"` + OsVersion *string `json:"os_version"` + ActivationID uuid.UUID `json:"activation_id"` +} + +func (q *Queries) RecordClientCheckin(ctx context.Context, arg RecordClientCheckinParams) error { + _, err := q.db.Exec(ctx, recordClientCheckin, + arg.OrganizationID, + arg.ProductID, + arg.LicenseID, + arg.Version, + arg.Build, + arg.Platform, + arg.Arch, + arg.OsVersion, + arg.ActivationID, + ) + return err +} diff --git a/backend/internal/db/models.go b/backend/internal/db/models.go index 27cf3a2..6c3c830 100644 --- a/backend/internal/db/models.go +++ b/backend/internal/db/models.go @@ -19,6 +19,12 @@ type Activation struct { LicenseID uuid.UUID `json:"license_id"` DeactivatedAt pgtype.Timestamptz `json:"deactivated_at"` DeactivationReason *string `json:"deactivation_reason"` + LastSeenAt pgtype.Timestamptz `json:"last_seen_at"` + CurrentVersion *string `json:"current_version"` + CurrentBuild *string `json:"current_build"` + Platform *string `json:"platform"` + Arch *string `json:"arch"` + OsVersion *string `json:"os_version"` } type AdminAuditLog struct { @@ -79,6 +85,14 @@ type ClientCheckin struct { CreatedAt pgtype.Timestamptz `json:"created_at"` } +type DailyVersionAdoption struct { + Date pgtype.Date `json:"date"` + OrganizationID uuid.UUID `json:"organization_id"` + ProductID uuid.UUID `json:"product_id"` + Version string `json:"version"` + DeviceCount int64 `json:"device_count"` +} + type Device struct { ID uuid.UUID `json:"id"` HwidHash []byte `json:"hwid_hash"` diff --git a/backend/internal/db/updates.sql.go b/backend/internal/db/updates.sql.go index 2e3b2ef..e6388cc 100644 --- a/backend/internal/db/updates.sql.go +++ b/backend/internal/db/updates.sql.go @@ -156,19 +156,6 @@ func (q *Queries) CreateUpdateChannel(ctx context.Context, arg CreateUpdateChann return i, err } -const deleteExpiredUpdateChecks = `-- name: DeleteExpiredUpdateChecks :execrows -DELETE FROM update_checks -WHERE created_at < now() - make_interval(days => $1::int) -` - -func (q *Queries) DeleteExpiredUpdateChecks(ctx context.Context, retentionDays int32) (int64, error) { - result, err := q.db.Exec(ctx, deleteExpiredUpdateChecks, retentionDays) - if err != nil { - return 0, err - } - return result.RowsAffected(), nil -} - const deleteProductUpdateConfig = `-- name: DeleteProductUpdateConfig :one DELETE FROM product_update_configs WHERE id = $1 AND organization_id = $2 diff --git a/backend/internal/features/clientsync/service.go b/backend/internal/features/clientsync/service.go index c4075d2..c02f4c6 100644 --- a/backend/internal/features/clientsync/service.go +++ b/backend/internal/features/clientsync/service.go @@ -13,10 +13,14 @@ import ( type Service struct { validation *validation.Service updates *update.Service - recorder interface{ Record(diagnostics.Checkin) } + recorder interface { + Record(context.Context, diagnostics.Checkin) error + } } -func NewService(validationSvc *validation.Service, updateSvc *update.Service, recorder interface{ Record(diagnostics.Checkin) }) *Service { +func NewService(validationSvc *validation.Service, updateSvc *update.Service, recorder interface { + Record(context.Context, diagnostics.Checkin) error +}) *Service { return &Service{validation: validationSvc, updates: updateSvc, recorder: recorder} } @@ -26,8 +30,8 @@ func (svc *Service) Sync(ctx context.Context, token string, req Request) (Respon return Response{}, err } version := strings.TrimSpace(req.Version) - if version != "" && svc.recorder != nil { - svc.recorder.Record(diagnostics.Checkin{ + if svc.recorder != nil { + if err := svc.recorder.Record(ctx, diagnostics.Checkin{ OrganizationID: auth.OrganizationID, ProductID: auth.License.ProductID, LicenseID: auth.LicenseID, @@ -37,7 +41,9 @@ func (svc *Service) Sync(ctx context.Context, token string, req Request) (Respon Platform: req.Platform, Arch: req.Arch, OSVersion: req.OSVersion, - }) + }); err != nil { + return Response{}, err + } } refreshed, err := svc.validation.Refresh(ctx, auth) if err != nil { diff --git a/backend/internal/features/diagnostics/recorder.go b/backend/internal/features/diagnostics/recorder.go index 62842db..9c8dbdb 100644 --- a/backend/internal/features/diagnostics/recorder.go +++ b/backend/internal/features/diagnostics/recorder.go @@ -4,86 +4,63 @@ import ( "context" "log/slog" "sync" - "sync/atomic" "time" "github.com/cheetahbyte/clave/internal/observability" ) type checkinStore interface { - InsertCheckin(context.Context, Checkin) error - DeleteExpiredCheckins(context.Context, int) (int64, error) + RecordCheckin(context.Context, Checkin) error + AggregateClosedDatesAndCleanup(context.Context, int) (int64, error) } type Recorder struct { store checkinStore retentionDays int - records chan Checkin cancel context.CancelFunc wg sync.WaitGroup - dropped atomic.Uint64 } -func NewRecorder(store checkinStore, retentionDays, capacity int) *Recorder { - if capacity < 1 { - capacity = 256 - } +func NewRecorder(store checkinStore, retentionDays int) *Recorder { ctx, cancel := context.WithCancel(context.Background()) - recorder := &Recorder{ - store: store, retentionDays: retentionDays, - records: make(chan Checkin, capacity), cancel: cancel, - } + recorder := &Recorder{store: store, retentionDays: retentionDays, cancel: cancel} recorder.wg.Add(1) go recorder.run(ctx) return recorder } -func (r *Recorder) Record(checkin Checkin) { - select { - case r.records <- checkin: - default: - r.dropped.Add(1) - observability.CountClientCheckinTelemetry(context.Background(), "dropped") +func (r *Recorder) Record(ctx context.Context, checkin Checkin) error { + if err := r.store.RecordCheckin(ctx, checkin); err != nil { + observability.CountClientCheckinTelemetry(ctx, "error") + return err } + observability.CountClientCheckinTelemetry(ctx, "recorded") + return nil } func (r *Recorder) run(ctx context.Context) { defer r.wg.Done() - var cleanup <-chan time.Time - if r.retentionDays > 0 { - ticker := time.NewTicker(24 * time.Hour) - defer ticker.Stop() - cleanup = ticker.C - r.cleanup(ctx) - } + ticker := time.NewTicker(24 * time.Hour) + defer ticker.Stop() + r.aggregateAndCleanup(ctx) for { select { - case record, ok := <-r.records: - if !ok { - return - } - if err := r.store.InsertCheckin(ctx, record); err != nil { - slog.Warn("failed to record client check-in", "err", err) - observability.CountClientCheckinTelemetry(context.Background(), "error") - } else { - observability.CountClientCheckinTelemetry(context.Background(), "recorded") - } - case <-cleanup: - r.cleanup(ctx) + case <-ticker.C: + r.aggregateAndCleanup(ctx) case <-ctx.Done(): return } } } -func (r *Recorder) cleanup(ctx context.Context) { - if _, err := r.store.DeleteExpiredCheckins(ctx, r.retentionDays); err != nil { - slog.Warn("failed to clean expired client check-ins", "err", err) +func (r *Recorder) aggregateAndCleanup(ctx context.Context) { + if _, err := r.store.AggregateClosedDatesAndCleanup(ctx, r.retentionDays); err != nil && ctx.Err() == nil { + slog.Warn("failed to aggregate or clean client check-ins", "err", err) } } func (r *Recorder) Close(ctx context.Context) { - close(r.records) + r.cancel() done := make(chan struct{}) go func() { r.wg.Wait() @@ -92,8 +69,5 @@ func (r *Recorder) Close(ctx context.Context) { select { case <-done: case <-ctx.Done(): - r.cancel() - <-done } - r.cancel() } diff --git a/backend/internal/features/diagnostics/recorder_test.go b/backend/internal/features/diagnostics/recorder_test.go index 8c5cbab..2c08f03 100644 --- a/backend/internal/features/diagnostics/recorder_test.go +++ b/backend/internal/features/diagnostics/recorder_test.go @@ -2,50 +2,63 @@ package diagnostics import ( "context" - "sync/atomic" + "errors" "testing" "time" ) -type blockingCheckinStore struct { - started chan struct{} - release chan struct{} - inserts atomic.Int32 +type checkinStoreStub struct { + recordErr error + recorded chan Checkin + cleanupCalled chan int } -func (s *blockingCheckinStore) InsertCheckin(context.Context, Checkin) error { - if s.inserts.Add(1) == 1 { - close(s.started) +func (s *checkinStoreStub) RecordCheckin(_ context.Context, checkin Checkin) error { + if s.recorded != nil { + s.recorded <- checkin } - <-s.release - return nil + return s.recordErr } -func (*blockingCheckinStore) DeleteExpiredCheckins(context.Context, int) (int64, error) { +func (s *checkinStoreStub) AggregateClosedDatesAndCleanup(_ context.Context, retentionDays int) (int64, error) { + if s.cleanupCalled != nil { + s.cleanupCalled <- retentionDays + } return 0, nil } -func TestRecorderDropsWhenFullWithoutBlocking(t *testing.T) { - store := &blockingCheckinStore{started: make(chan struct{}), release: make(chan struct{})} - recorder := NewRecorder(store, 0, 1) - recorder.Record(Checkin{}) - <-store.started - recorder.Record(Checkin{}) - - start := time.Now() - recorder.Record(Checkin{}) - if elapsed := time.Since(start); elapsed > 50*time.Millisecond { - t.Fatalf("full recorder blocked request path for %s", elapsed) +func TestRecorderPersistsCheckinSynchronously(t *testing.T) { + store := &checkinStoreStub{recorded: make(chan Checkin, 1)} + recorder := NewRecorder(store, 7) + want := Checkin{Version: "2.0.0"} + if err := recorder.Record(t.Context(), want); err != nil { + t.Fatal(err) + } + if got := <-store.recorded; got.Version != want.Version { + t.Fatalf("recorded version = %q, want %q", got.Version, want.Version) } - if got := recorder.dropped.Load(); got != 1 { - t.Fatalf("dropped = %d, want 1", got) + recorder.Close(t.Context()) +} + +func TestRecorderReturnsWriteError(t *testing.T) { + want := errors.New("database unavailable") + recorder := NewRecorder(&checkinStoreStub{recordErr: want}, 7) + if err := recorder.Record(t.Context(), Checkin{}); !errors.Is(err, want) { + t.Fatalf("error = %v, want %v", err, want) } + recorder.Close(t.Context()) +} - close(store.release) - ctx, cancel := context.WithTimeout(context.Background(), time.Second) - defer cancel() - recorder.Close(ctx) - if got := store.inserts.Load(); got != 2 { - t.Fatalf("inserts = %d, want 2", got) +func TestRecorderRunsDailyWorkAtStartupWithConfiguredRetention(t *testing.T) { + called := make(chan int, 1) + recorder := NewRecorder(&checkinStoreStub{cleanupCalled: called}, 7) + select { + case got := <-called: + if got != 7 { + t.Fatalf("retention = %d, want 7", got) + } + case <-time.After(time.Second): + t.Fatal("startup aggregation was not called") } + recorder.Close(t.Context()) } diff --git a/backend/internal/features/diagnostics/repository.go b/backend/internal/features/diagnostics/repository.go index 4e75f0f..c5e87d3 100644 --- a/backend/internal/features/diagnostics/repository.go +++ b/backend/internal/features/diagnostics/repository.go @@ -2,19 +2,23 @@ package diagnostics import ( "context" + "fmt" + "math" "strings" "github.com/cheetahbyte/clave/internal/db" "github.com/google/uuid" "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" ) type Repository struct { - q *db.Queries + q *db.Queries + pool *pgxpool.Pool } -func NewRepository(q *db.Queries) *Repository { - return &Repository{q: q} +func NewRepository(q *db.Queries, pool *pgxpool.Pool) *Repository { + return &Repository{q: q, pool: pool} } func optionalString(value string) *string { @@ -25,8 +29,8 @@ func optionalString(value string) *string { return &value } -func (r *Repository) InsertCheckin(ctx context.Context, checkin Checkin) error { - _, err := r.q.InsertClientCheckin(ctx, db.InsertClientCheckinParams{ +func (r *Repository) RecordCheckin(ctx context.Context, checkin Checkin) error { + return r.q.RecordClientCheckin(ctx, db.RecordClientCheckinParams{ OrganizationID: checkin.OrganizationID, ProductID: checkin.ProductID, LicenseID: checkin.LicenseID, @@ -37,15 +41,49 @@ func (r *Repository) InsertCheckin(ctx context.Context, checkin Checkin) error { Arch: optionalString(checkin.Arch), OsVersion: optionalString(checkin.OSVersion), }) - return err } -func (r *Repository) DeleteExpiredCheckins(ctx context.Context, retentionDays int) (int64, error) { - return r.q.DeleteExpiredClientCheckins(ctx, int32(retentionDays)) +// AggregateClosedDatesAndCleanup recomputes every closed UTC date still in raw +// storage before deleting expired rows. The transaction makes retries safe and +// ensures raw data is only deleted after its aggregate was persisted. +func (r *Repository) AggregateClosedDatesAndCleanup(ctx context.Context, retentionDays int) (int64, error) { + tx, err := r.pool.Begin(ctx) + if err != nil { + return 0, err + } + defer tx.Rollback(ctx) //nolint:errcheck + qtx := r.q.WithTx(tx) + + dates, err := qtx.ListClosedClientCheckinDates(ctx) + if err != nil { + return 0, err + } + for _, date := range dates { + if err := qtx.DeleteDailyVersionAdoptionForDate(ctx, date); err != nil { + return 0, err + } + if err := qtx.InsertDailyVersionAdoptionForDate(ctx, date); err != nil { + return 0, err + } + } + + if retentionDays < 0 || retentionDays > math.MaxInt32 { + return 0, fmt.Errorf("retentionDays must be between 0 and %d", math.MaxInt32) + } + retentionDays32 := int32(retentionDays) + + deleted, err := qtx.DeleteExpiredClientCheckins(ctx, retentionDays32) + if err != nil { + return 0, err + } + if err := tx.Commit(ctx); err != nil { + return 0, err + } + return deleted, nil } -func (r *Repository) ListLatestCheckins(ctx context.Context, orgID uuid.UUID, productID pgtype.UUID, days int) ([]LatestCheckin, error) { - rows, err := r.q.ListLatestClientCheckins(ctx, db.ListLatestClientCheckinsParams{ +func (r *Repository) ListCurrentStates(ctx context.Context, orgID uuid.UUID, productID pgtype.UUID, days int) ([]LatestCheckin, error) { + rows, err := r.q.ListCurrentClientStates(ctx, db.ListCurrentClientStatesParams{ OrganizationID: orgID, Days: int32(days), ProductID: productID, @@ -58,19 +96,26 @@ func (r *Repository) ListLatestCheckins(ctx context.Context, orgID uuid.UUID, pr result = append(result, LatestCheckin{ ActivationID: row.ActivationID, Hostname: row.Hostname, - Version: row.Version, + Version: valueOrEmpty(row.Version), Build: row.Build, Platform: row.Platform, Arch: row.Arch, OSVersion: row.OsVersion, - CreatedAt: row.CreatedAt.Time, + CreatedAt: row.LastSeenAt.Time, }) } return result, nil } +func valueOrEmpty(value *string) string { + if value == nil { + return "" + } + return *value +} + func (r *Repository) ListDailyVersions(ctx context.Context, orgID uuid.UUID, productID pgtype.UUID, days int) ([]DailyVersion, error) { - rows, err := r.q.ListDailyLatestClientVersions(ctx, db.ListDailyLatestClientVersionsParams{ + rows, err := r.q.ListDailyVersionAdoption(ctx, db.ListDailyVersionAdoptionParams{ OrganizationID: orgID, Days: int32(days), ProductID: productID, diff --git a/backend/internal/features/diagnostics/service.go b/backend/internal/features/diagnostics/service.go index 52fc40f..19bf61f 100644 --- a/backend/internal/features/diagnostics/service.go +++ b/backend/internal/features/diagnostics/service.go @@ -11,7 +11,7 @@ import ( ) type adoptionRepository interface { - ListLatestCheckins(context.Context, uuid.UUID, pgtype.UUID, int) ([]LatestCheckin, error) + ListCurrentStates(context.Context, uuid.UUID, pgtype.UUID, int) ([]LatestCheckin, error) ListDailyVersions(context.Context, uuid.UUID, pgtype.UUID, int) ([]DailyVersion, error) } @@ -29,7 +29,7 @@ func (svc *Service) VersionAdoption(ctx context.Context, orgID uuid.UUID, produc group, groupCtx := errgroup.WithContext(ctx) group.Go(func() error { var err error - latest, err = svc.repo.ListLatestCheckins(groupCtx, orgID, productID, days) + latest, err = svc.repo.ListCurrentStates(groupCtx, orgID, productID, days) return err }) group.Go(func() error { diff --git a/backend/internal/features/diagnostics/service_test.go b/backend/internal/features/diagnostics/service_test.go index 2720899..ba9d8db 100644 --- a/backend/internal/features/diagnostics/service_test.go +++ b/backend/internal/features/diagnostics/service_test.go @@ -17,7 +17,7 @@ type fakeAdoptionRepository struct { dailyErr error } -func (f *fakeAdoptionRepository) ListLatestCheckins(context.Context, uuid.UUID, pgtype.UUID, int) ([]LatestCheckin, error) { +func (f *fakeAdoptionRepository) ListCurrentStates(context.Context, uuid.UUID, pgtype.UUID, int) ([]LatestCheckin, error) { return f.latest, f.latestErr } diff --git a/backend/internal/features/update/check_recorder.go b/backend/internal/features/update/check_recorder.go index f60a32b..c77e972 100644 --- a/backend/internal/features/update/check_recorder.go +++ b/backend/internal/features/update/check_recorder.go @@ -5,7 +5,6 @@ import ( "log/slog" "sync" "sync/atomic" - "time" "github.com/cheetahbyte/clave/internal/observability" "github.com/google/uuid" @@ -20,24 +19,22 @@ type UpdateCheckRecord struct { type checkRecordStore interface { InsertUpdateCheck(context.Context, uuid.UUID, uuid.UUID, uuid.UUID, string, string, string, string, string, string, string, string, *uuid.UUID) error - DeleteExpiredUpdateChecks(context.Context, int) (int64, error) } type UpdateCheckRecorder struct { - store checkRecordStore - retentionDays int - records chan UpdateCheckRecord - cancel context.CancelFunc - wg sync.WaitGroup - dropped atomic.Uint64 + store checkRecordStore + records chan UpdateCheckRecord + cancel context.CancelFunc + wg sync.WaitGroup + dropped atomic.Uint64 } -func NewUpdateCheckRecorder(store checkRecordStore, retentionDays, capacity int) *UpdateCheckRecorder { +func NewUpdateCheckRecorder(store checkRecordStore, capacity int) *UpdateCheckRecorder { if capacity < 1 { capacity = 256 } ctx, cancel := context.WithCancel(context.Background()) - r := &UpdateCheckRecorder{store: store, retentionDays: retentionDays, records: make(chan UpdateCheckRecord, capacity), cancel: cancel} + r := &UpdateCheckRecorder{store: store, records: make(chan UpdateCheckRecord, capacity), cancel: cancel} r.wg.Add(1) go r.run(ctx) return r @@ -54,13 +51,6 @@ func (r *UpdateCheckRecorder) Record(record UpdateCheckRecord) { func (r *UpdateCheckRecorder) run(ctx context.Context) { defer r.wg.Done() - var cleanup <-chan time.Time - if r.retentionDays > 0 { - ticker := time.NewTicker(24 * time.Hour) - defer ticker.Stop() - cleanup = ticker.C - r.cleanup(ctx) - } for { select { case record, ok := <-r.records: @@ -75,20 +65,12 @@ func (r *UpdateCheckRecorder) run(ctx context.Context) { } else { observability.CountUpdateCheckTelemetry(context.Background(), "recorded") } - case <-cleanup: - r.cleanup(ctx) case <-ctx.Done(): return } } } -func (r *UpdateCheckRecorder) cleanup(ctx context.Context) { - if _, err := r.store.DeleteExpiredUpdateChecks(ctx, r.retentionDays); err != nil { - slog.Warn("failed to clean old update checks", "err", err) - } -} - func (r *UpdateCheckRecorder) Close(ctx context.Context) { close(r.records) done := make(chan struct{}) diff --git a/backend/internal/features/update/check_recorder_test.go b/backend/internal/features/update/check_recorder_test.go index 9f1989b..e725093 100644 --- a/backend/internal/features/update/check_recorder_test.go +++ b/backend/internal/features/update/check_recorder_test.go @@ -22,13 +22,9 @@ func (s *blockingRecordStore) InsertUpdateCheck(context.Context, uuid.UUID, uuid <-s.release return nil } -func (*blockingRecordStore) DeleteExpiredUpdateChecks(context.Context, int) (int64, error) { - return 0, nil -} - func TestUpdateCheckRecorderDropsWhenFullWithoutBlocking(t *testing.T) { store := &blockingRecordStore{started: make(chan struct{}), release: make(chan struct{})} - recorder := NewUpdateCheckRecorder(store, 0, 1) + recorder := NewUpdateCheckRecorder(store, 1) recorder.Record(UpdateCheckRecord{}) <-store.started recorder.Record(UpdateCheckRecord{}) diff --git a/backend/internal/features/update/repository.go b/backend/internal/features/update/repository.go index b0f6f68..7c85163 100644 --- a/backend/internal/features/update/repository.go +++ b/backend/internal/features/update/repository.go @@ -416,10 +416,6 @@ func (r *Repository) InsertUpdateCheck(ctx context.Context, orgID, productID, li return err } -func (r *Repository) DeleteExpiredUpdateChecks(ctx context.Context, retentionDays int) (int64, error) { - return r.q.DeleteExpiredUpdateChecks(ctx, int32(retentionDays)) -} - func MustParseProviderConfig(raw []byte) map[string]any { if len(raw) == 0 { return nil diff --git a/backend/internal/observability/business.go b/backend/internal/observability/business.go index 600b624..3c19f0d 100644 --- a/backend/internal/observability/business.go +++ b/backend/internal/observability/business.go @@ -17,6 +17,7 @@ var ( bizAuditEvents metric.Int64Counter bizUpdateCheckTelemetry metric.Int64Counter bizClientCheckinTelemetry metric.Int64Counter + bizRetentionCleanup metric.Int64Counter bizDeltaJobs metric.Int64Counter bizDeltaPatchRatio metric.Float64Histogram ) @@ -75,6 +76,11 @@ func initBusinessMetrics(m metric.Meter) { if err != nil { panic(err) } + bizRetentionCleanup, err = m.Int64Counter("biz.retention_cleanup_runs_total", + metric.WithDescription("Retention cleanup operation outcomes")) + if err != nil { + panic(err) + } bizDeltaJobs, err = m.Int64Counter("biz.delta_jobs_total", metric.WithDescription("Delta job state transitions")) if err != nil { panic(err) @@ -85,6 +91,15 @@ func initBusinessMetrics(m metric.Meter) { } } +func CountRetentionCleanup(ctx context.Context, dataset, outcome string) { + if bizRetentionCleanup != nil { + bizRetentionCleanup.Add(ctx, 1, metric.WithAttributes( + attribute.String("dataset", dataset), + attribute.String("outcome", outcome), + )) + } +} + func CountDeltaJob(ctx context.Context, status string) { if bizDeltaJobs != nil { bizDeltaJobs.Add(ctx, 1, metric.WithAttributes(attribute.String("status", status))) diff --git a/backend/internal/retention/worker.go b/backend/internal/retention/worker.go new file mode 100644 index 0000000..13cb6a4 --- /dev/null +++ b/backend/internal/retention/worker.go @@ -0,0 +1,137 @@ +package retention + +import ( + "context" + "log/slog" + "math" + "sync" + "time" + + "github.com/cheetahbyte/clave/internal/db" + "github.com/cheetahbyte/clave/internal/observability" +) + +const ( + cleanupInterval = 24 * time.Hour + operationTimeout = 30 * time.Second +) + +type Policies struct { + AuditMetadataDays int + AuditLogDays int + UpdateCheckDays int +} + +type store interface { + DeleteStaleSelfServiceTokens(context.Context) (int64, error) + DeleteStaleOrganizationInvites(context.Context) (int64, error) + DeleteStaleAdminEmailCodes(context.Context) (int64, error) + ScrubStaleAuditSecurityMetadata(context.Context, int32) (int64, error) + DeleteStaleAuditLogs(context.Context, int32) (int64, error) + DeleteStaleUpdateChecks(context.Context, int32) (int64, error) +} + +type Worker struct { + store store + policies Policies + cancel context.CancelFunc + wg sync.WaitGroup +} + +func NewWorker(q *db.Queries, policies Policies) *Worker { + return newWorker(q, policies) +} + +func newWorker(store store, policies Policies) *Worker { + ctx, cancel := context.WithCancel(context.Background()) + worker := &Worker{store: store, policies: policies, cancel: cancel} + worker.wg.Add(1) + go worker.run(ctx) + return worker +} + +func (w *Worker) run(ctx context.Context) { + defer w.wg.Done() + ticker := time.NewTicker(cleanupInterval) + defer ticker.Stop() + w.cleanup(ctx) + for { + select { + case <-ticker.C: + w.cleanup(ctx) + case <-ctx.Done(): + return + } + } +} + +func toInt32(value int) (int32, bool) { + if value < math.MinInt32 || value > math.MaxInt32 { + return 0, false + } + return int32(value), true +} + +func (w *Worker) cleanup(ctx context.Context) { + auditMetadataDays, ok := toInt32(w.policies.AuditMetadataDays) + if !ok { + slog.Warn("retention cleanup skipped due to invalid policy value", "policy", "AuditMetadataDays", "value", w.policies.AuditMetadataDays) + return + } + auditLogDays, ok := toInt32(w.policies.AuditLogDays) + if !ok { + slog.Warn("retention cleanup skipped due to invalid policy value", "policy", "AuditLogDays", "value", w.policies.AuditLogDays) + return + } + updateCheckDays, ok := toInt32(w.policies.UpdateCheckDays) + if !ok { + slog.Warn("retention cleanup skipped due to invalid policy value", "policy", "UpdateCheckDays", "value", w.policies.UpdateCheckDays) + return + } + + operations := []struct { + name string + run func(context.Context) (int64, error) + }{ + {"self_service_tokens", w.store.DeleteStaleSelfServiceTokens}, + {"organization_invites", w.store.DeleteStaleOrganizationInvites}, + {"admin_email_codes", w.store.DeleteStaleAdminEmailCodes}, + {"audit_security_metadata", func(ctx context.Context) (int64, error) { + return w.store.ScrubStaleAuditSecurityMetadata(ctx, auditMetadataDays) + }}, + {"audit_logs", func(ctx context.Context) (int64, error) { + return w.store.DeleteStaleAuditLogs(ctx, auditLogDays) + }}, + {"update_checks", func(ctx context.Context) (int64, error) { + return w.store.DeleteStaleUpdateChecks(ctx, updateCheckDays) + }}, + } + for _, operation := range operations { + if ctx.Err() != nil { + return + } + operationCtx, cancel := context.WithTimeout(ctx, operationTimeout) + rows, err := operation.run(operationCtx) + cancel() + if err != nil { + slog.Warn("retention cleanup failed", "dataset", operation.name, "err", err) + observability.CountRetentionCleanup(ctx, operation.name, "error") + continue + } + slog.Info("retention cleanup completed", "dataset", operation.name, "rows", rows) + observability.CountRetentionCleanup(ctx, operation.name, "success") + } +} + +func (w *Worker) Close(ctx context.Context) { + w.cancel() + done := make(chan struct{}) + go func() { + w.wg.Wait() + close(done) + }() + select { + case <-done: + case <-ctx.Done(): + } +} diff --git a/backend/internal/retention/worker_test.go b/backend/internal/retention/worker_test.go new file mode 100644 index 0000000..1f81adf --- /dev/null +++ b/backend/internal/retention/worker_test.go @@ -0,0 +1,60 @@ +package retention + +import ( + "context" + "errors" + "testing" +) + +type storeStub struct { + calls []string + inviteErr error + metadataDays int32 + auditDays int32 + updateDays int32 +} + +func (s *storeStub) called(name string, err error) (int64, error) { + s.calls = append(s.calls, name) + return 2, err +} + +func (s *storeStub) DeleteStaleSelfServiceTokens(context.Context) (int64, error) { + return s.called("self_service_tokens", nil) +} +func (s *storeStub) DeleteStaleOrganizationInvites(context.Context) (int64, error) { + return s.called("organization_invites", s.inviteErr) +} +func (s *storeStub) DeleteStaleAdminEmailCodes(context.Context) (int64, error) { + return s.called("admin_email_codes", nil) +} +func (s *storeStub) ScrubStaleAuditSecurityMetadata(_ context.Context, days int32) (int64, error) { + s.metadataDays = days + return s.called("audit_security_metadata", nil) +} +func (s *storeStub) DeleteStaleAuditLogs(_ context.Context, days int32) (int64, error) { + s.auditDays = days + return s.called("audit_logs", nil) +} +func (s *storeStub) DeleteStaleUpdateChecks(_ context.Context, days int32) (int64, error) { + s.updateDays = days + return s.called("update_checks", nil) +} + +func TestCleanupContinuesAfterFailureAndUsesPolicies(t *testing.T) { + store := &storeStub{inviteErr: errors.New("database unavailable")} + worker := &Worker{store: store, policies: Policies{ + AuditMetadataDays: 90, + AuditLogDays: 180, + UpdateCheckDays: 90, + }} + + worker.cleanup(t.Context()) + + if len(store.calls) != 6 { + t.Fatalf("calls = %v", store.calls) + } + if store.metadataDays != 90 || store.auditDays != 180 || store.updateDays != 90 { + t.Fatalf("policy days = metadata:%d audit:%d update:%d", store.metadataDays, store.auditDays, store.updateDays) + } +} diff --git a/backend/internal/shared/events/publisher.go b/backend/internal/shared/events/publisher.go index ebd08e8..13cc625 100644 --- a/backend/internal/shared/events/publisher.go +++ b/backend/internal/shared/events/publisher.go @@ -6,6 +6,7 @@ import ( "context" "encoding/json" "fmt" + "strconv" "sync" "time" @@ -16,6 +17,13 @@ import ( // routed by their Type (e.g. "license.created"). const exchange = "clave.events" +const ( + mfaEmailTTL = 10 * time.Minute + magicLinkEmailTTL = 15 * time.Minute + inviteEmailTTL = 7 * 24 * time.Hour + licenseEmailTTL = 7 * 24 * time.Hour +) + // EmailEvent is the wire shape consumed by the emailer worker. It must stay in // sync with EmailEvent in emailer/src/templates.ts. type EmailEvent struct { @@ -99,12 +107,35 @@ func (p *Publisher) publish(ctx context.Context, routingKey string, event any) e pubCtx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() - return ch.PublishWithContext(pubCtx, exchange, routingKey, false, false, amqp.Publishing{ + publishing := amqp.Publishing{ ContentType: "application/json", DeliveryMode: amqp.Persistent, Timestamp: time.Now(), Body: body, - }) + } + if expiration := eventExpiration(event); expiration > 0 { + publishing.Expiration = strconv.FormatInt(expiration.Milliseconds(), 10) + } + return ch.PublishWithContext(pubCtx, exchange, routingKey, false, false, publishing) +} + +func eventExpiration(event any) time.Duration { + email, ok := event.(EmailEvent) + if !ok { + return 0 + } + switch email.Type { + case "admin.2fa_code": + return mfaEmailTTL + case "selfservice.magic_link": + return magicLinkEmailTTL + case "organization.invite": + return inviteEmailTTL + case "license.created", "license.replaced": + return licenseEmailTTL + default: + return 0 + } } // PublishLicenseCreated emits a "license.created" event for the emailer worker. diff --git a/backend/internal/shared/events/publisher_test.go b/backend/internal/shared/events/publisher_test.go new file mode 100644 index 0000000..1800326 --- /dev/null +++ b/backend/internal/shared/events/publisher_test.go @@ -0,0 +1,30 @@ +package events + +import ( + "testing" + "time" +) + +func TestEventExpirationMatchesCredentialLifecycle(t *testing.T) { + tests := []struct { + typeName string + want time.Duration + }{ + {"admin.2fa_code", 10 * time.Minute}, + {"selfservice.magic_link", 15 * time.Minute}, + {"organization.invite", 7 * 24 * time.Hour}, + {"license.created", 7 * 24 * time.Hour}, + {"license.replaced", 7 * 24 * time.Hour}, + {"unknown", 0}, + } + for _, test := range tests { + t.Run(test.typeName, func(t *testing.T) { + if got := eventExpiration(EmailEvent{Type: test.typeName}); got != test.want { + t.Fatalf("expiration = %s, want %s", got, test.want) + } + }) + } + if got := eventExpiration(DeltaGenerateEvent{}); got != 0 { + t.Fatalf("delta expiration = %s, want none", got) + } +} diff --git a/backend/migrations/20260721000002_checkin_data_lifecycle.sql b/backend/migrations/20260721000002_checkin_data_lifecycle.sql new file mode 100644 index 0000000..ee2b3e7 --- /dev/null +++ b/backend/migrations/20260721000002_checkin_data_lifecycle.sql @@ -0,0 +1,78 @@ +-- +goose Up +-- +goose StatementBegin + +ALTER TABLE activations + ADD COLUMN IF NOT EXISTS last_seen_at TIMESTAMPTZ, + ADD COLUMN IF NOT EXISTS current_version TEXT, + ADD COLUMN IF NOT EXISTS current_build TEXT, + ADD COLUMN IF NOT EXISTS platform TEXT, + ADD COLUMN IF NOT EXISTS arch TEXT, + ADD COLUMN IF NOT EXISTS os_version TEXT; + +CREATE INDEX IF NOT EXISTS idx_activations_current_state + ON activations (last_seen_at DESC) + WHERE deactivated_at IS NULL AND last_seen_at IS NOT NULL; + +CREATE TABLE IF NOT EXISTS daily_version_adoption ( + date DATE NOT NULL, + organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE, + product_id UUID NOT NULL REFERENCES products(id) ON DELETE CASCADE, + version TEXT NOT NULL, + device_count BIGINT NOT NULL CHECK (device_count >= 0), + PRIMARY KEY (date, organization_id, product_id, version) +); + +CREATE INDEX IF NOT EXISTS idx_daily_version_adoption_org_date + ON daily_version_adoption (organization_id, date DESC); + +WITH latest AS ( + SELECT DISTINCT ON (activation_id) + activation_id, version, build, platform, arch, os_version, created_at + FROM client_checkins + ORDER BY activation_id, created_at DESC, id DESC +) +UPDATE activations a +SET last_seen_at = latest.created_at, + current_version = latest.version, + current_build = latest.build, + platform = latest.platform, + arch = latest.arch, + os_version = latest.os_version +FROM latest +WHERE a.id = latest.activation_id; + +INSERT INTO daily_version_adoption (date, organization_id, product_id, version, device_count) +SELECT observed_date, organization_id, product_id, version, count(*) +FROM ( + SELECT DISTINCT ON (created_at::date, activation_id) + (created_at AT TIME ZONE 'UTC')::date AS observed_date, + organization_id, + product_id, + activation_id, + version + FROM client_checkins + WHERE (created_at AT TIME ZONE 'UTC')::date < (now() AT TIME ZONE 'UTC')::date + ORDER BY (created_at AT TIME ZONE 'UTC')::date, activation_id, created_at DESC, id DESC +) latest_daily +GROUP BY observed_date, organization_id, product_id, version +ON CONFLICT (date, organization_id, product_id, version) +DO UPDATE SET device_count = EXCLUDED.device_count; + +-- +goose StatementEnd + +-- +goose Down +-- +goose StatementBegin + +DROP INDEX IF EXISTS idx_daily_version_adoption_org_date; +DROP TABLE IF EXISTS daily_version_adoption; +DROP INDEX IF EXISTS idx_activations_current_state; + +ALTER TABLE activations + DROP COLUMN IF EXISTS os_version, + DROP COLUMN IF EXISTS arch, + DROP COLUMN IF EXISTS platform, + DROP COLUMN IF EXISTS current_build, + DROP COLUMN IF EXISTS current_version, + DROP COLUMN IF EXISTS last_seen_at; + +-- +goose StatementEnd diff --git a/backend/migrations/20260721000003_retention_indexes.sql b/backend/migrations/20260721000003_retention_indexes.sql new file mode 100644 index 0000000..5c2c094 --- /dev/null +++ b/backend/migrations/20260721000003_retention_indexes.sql @@ -0,0 +1,33 @@ +-- +goose Up +-- +goose StatementBegin + +CREATE INDEX IF NOT EXISTS idx_self_service_tokens_used + ON self_service_tokens (used_at) + WHERE used_at IS NOT NULL; + +CREATE INDEX IF NOT EXISTS idx_organization_invites_expiry + ON organization_invites (expires_at); + +CREATE INDEX IF NOT EXISTS idx_organization_invites_accepted + ON organization_invites (accepted_at) + WHERE accepted_at IS NOT NULL; + +CREATE INDEX IF NOT EXISTS idx_admin_email_codes_expiry + ON admin_email_codes (expires_at); + +CREATE INDEX IF NOT EXISTS idx_admin_email_codes_used + ON admin_email_codes (used_at) + WHERE used_at IS NOT NULL; + +-- +goose StatementEnd + +-- +goose Down +-- +goose StatementBegin + +DROP INDEX IF EXISTS idx_admin_email_codes_used; +DROP INDEX IF EXISTS idx_admin_email_codes_expiry; +DROP INDEX IF EXISTS idx_organization_invites_accepted; +DROP INDEX IF EXISTS idx_organization_invites_expiry; +DROP INDEX IF EXISTS idx_self_service_tokens_used; + +-- +goose StatementEnd diff --git a/backend/queries/admin.sql b/backend/queries/admin.sql index 8fbebde..7d55367 100644 --- a/backend/queries/admin.sql +++ b/backend/queries/admin.sql @@ -42,10 +42,6 @@ SET used_at = now() WHERE admin_user_id = sqlc.arg(admin_user_id) AND used_at IS NULL; --- name: DeleteExpiredAdminEmailCodes :exec -DELETE FROM admin_email_codes -WHERE expires_at < now() - INTERVAL '1 day'; - -- name: InsertAuditLog :exec INSERT INTO admin_audit_log (admin_user_id, organization_id, action, resource_type, resource_id, metadata, ip, user_agent) VALUES ($1, $2, $3, $4, $5, $6, $7, $8); @@ -72,3 +68,36 @@ WHERE organization_id = $1 AND admin_user_id = $2; -- name: GetOrganizationBySlug :one SELECT * FROM organizations WHERE slug = $1; + +-- Retention maintenance queries are centralized here so one generated query +-- adapter can run the database lifecycle worker. + +-- name: DeleteStaleSelfServiceTokens :execrows +DELETE FROM self_service_tokens +WHERE expires_at < now() - INTERVAL '1 day' + OR used_at < now() - INTERVAL '1 day'; + +-- name: DeleteStaleOrganizationInvites :execrows +DELETE FROM organization_invites +WHERE (accepted_at IS NOT NULL AND accepted_at < now() - INTERVAL '30 days') + OR (accepted_at IS NULL AND expires_at < now() - INTERVAL '30 days'); + +-- name: DeleteStaleAdminEmailCodes :execrows +DELETE FROM admin_email_codes +WHERE expires_at < now() - INTERVAL '1 day' + OR used_at < now() - INTERVAL '1 day'; + +-- name: ScrubStaleAuditSecurityMetadata :execrows +UPDATE admin_audit_log +SET ip = NULL, + user_agent = NULL +WHERE created_at < now() - make_interval(days => sqlc.arg('retention_days')::int) + AND (ip IS NOT NULL OR user_agent IS NOT NULL); + +-- name: DeleteStaleAuditLogs :execrows +DELETE FROM admin_audit_log +WHERE created_at < now() - make_interval(days => sqlc.arg('retention_days')::int); + +-- name: DeleteStaleUpdateChecks :execrows +DELETE FROM update_checks +WHERE created_at < now() - make_interval(days => sqlc.arg('retention_days')::int); diff --git a/backend/queries/diagnostics.sql b/backend/queries/diagnostics.sql index f785cba..ee7bc76 100644 --- a/backend/queries/diagnostics.sql +++ b/backend/queries/diagnostics.sql @@ -1,64 +1,89 @@ --- name: InsertClientCheckin :one +-- name: RecordClientCheckin :exec +WITH updated AS ( + UPDATE activations a + SET last_seen_at = now(), + current_version = CASE WHEN sqlc.arg('version') <> '' THEN sqlc.arg('version') ELSE a.current_version END, + current_build = CASE WHEN sqlc.arg('version') <> '' THEN sqlc.narg('build') ELSE a.current_build END, + platform = CASE WHEN sqlc.arg('version') <> '' THEN sqlc.narg('platform') ELSE a.platform END, + arch = CASE WHEN sqlc.arg('version') <> '' THEN sqlc.narg('arch') ELSE a.arch END, + os_version = CASE WHEN sqlc.arg('version') <> '' THEN sqlc.narg('os_version') ELSE a.os_version END + WHERE a.id = sqlc.arg('activation_id') + AND a.license_id = sqlc.arg('license_id') + AND a.deactivated_at IS NULL + RETURNING a.id +) INSERT INTO client_checkins ( organization_id, product_id, license_id, activation_id, version, build, platform, arch, os_version ) -VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) -RETURNING *; +SELECT + sqlc.arg('organization_id'), sqlc.arg('product_id'), sqlc.arg('license_id'), updated.id, + sqlc.arg('version'), sqlc.narg('build'), sqlc.narg('platform'), + sqlc.narg('arch'), sqlc.narg('os_version') +FROM updated +WHERE sqlc.arg('version') <> ''; + +-- name: ListClosedClientCheckinDates :many +SELECT DISTINCT (created_at AT TIME ZONE 'UTC')::date AS date +FROM client_checkins +WHERE (created_at AT TIME ZONE 'UTC')::date < (now() AT TIME ZONE 'UTC')::date +ORDER BY date; + +-- name: DeleteDailyVersionAdoptionForDate :exec +DELETE FROM daily_version_adoption +WHERE date = sqlc.arg('date')::date; + +-- name: InsertDailyVersionAdoptionForDate :exec +INSERT INTO daily_version_adoption (date, organization_id, product_id, version, device_count) +SELECT observed_date, organization_id, product_id, version, count(*) +FROM ( + SELECT DISTINCT ON (activation_id) + (created_at AT TIME ZONE 'UTC')::date AS observed_date, + organization_id, + product_id, + activation_id, + version + FROM client_checkins + WHERE (created_at AT TIME ZONE 'UTC')::date = sqlc.arg('date')::date + ORDER BY activation_id, created_at DESC, id DESC +) latest_daily +GROUP BY observed_date, organization_id, product_id, version; -- name: DeleteExpiredClientCheckins :execrows DELETE FROM client_checkins WHERE created_at < now() - make_interval(days => sqlc.arg('retention_days')::int); --- name: ListLatestClientCheckins :many -WITH ranked AS ( - SELECT - c.*, - row_number() OVER ( - PARTITION BY c.activation_id - ORDER BY c.created_at DESC, c.id DESC - ) AS position - FROM client_checkins c - WHERE c.organization_id = sqlc.arg('organization_id')::uuid - AND c.created_at >= now() - make_interval(days => sqlc.arg('days')::int) - AND ( - sqlc.narg('product_id')::uuid IS NULL - OR c.product_id = sqlc.narg('product_id')::uuid - ) -) +-- name: ListCurrentClientStates :many SELECT - ranked.activation_id, + a.id AS activation_id, d.hostname, - ranked.version, - ranked.build, - ranked.platform, - ranked.arch, - ranked.os_version, - ranked.created_at -FROM ranked -JOIN activations a ON a.id = ranked.activation_id + a.current_version AS version, + a.current_build AS build, + a.platform, + a.arch, + a.os_version, + a.last_seen_at +FROM activations a JOIN devices d ON d.id = a.device_id -WHERE ranked.position = 1 +JOIN licenses l ON l.id = a.license_id +WHERE l.organization_id = sqlc.arg('organization_id')::uuid AND a.deactivated_at IS NULL -ORDER BY ranked.created_at DESC; + AND a.last_seen_at >= now() - make_interval(days => sqlc.arg('days')::int) + AND a.current_version IS NOT NULL + AND ( + sqlc.narg('product_id')::uuid IS NULL + OR l.product_id = sqlc.narg('product_id')::uuid + ) +ORDER BY a.last_seen_at DESC, a.id; --- name: ListDailyLatestClientVersions :many -SELECT date, version, count(*)::bigint AS device_count -FROM ( - SELECT DISTINCT ON (c.created_at::date, c.activation_id) - c.created_at::date AS date, - c.activation_id, - c.version - FROM client_checkins c - JOIN activations a ON a.id = c.activation_id - WHERE c.organization_id = sqlc.arg('organization_id')::uuid - AND c.created_at >= now() - make_interval(days => sqlc.arg('days')::int) - AND ( - sqlc.narg('product_id')::uuid IS NULL - OR c.product_id = sqlc.narg('product_id')::uuid - ) - AND a.deactivated_at IS NULL - ORDER BY c.created_at::date, c.activation_id, c.created_at DESC, c.id DESC -) latest +-- name: ListDailyVersionAdoption :many +SELECT date, version, sum(device_count)::bigint AS device_count +FROM daily_version_adoption +WHERE organization_id = sqlc.arg('organization_id')::uuid + AND date >= ((now() AT TIME ZONE 'UTC')::date - (sqlc.arg('days')::int - 1)) + AND ( + sqlc.narg('product_id')::uuid IS NULL + OR product_id = sqlc.narg('product_id')::uuid + ) GROUP BY date, version ORDER BY date, version; diff --git a/backend/queries/updates.sql b/backend/queries/updates.sql index ce437d5..9c0b4cf 100644 --- a/backend/queries/updates.sql +++ b/backend/queries/updates.sql @@ -165,10 +165,6 @@ WHERE release_id = $1; SELECT * FROM update_release_policies WHERE release_id = ANY($1::uuid[]); --- name: DeleteExpiredUpdateChecks :execrows -DELETE FROM update_checks -WHERE created_at < now() - make_interval(days => sqlc.arg('retention_days')::int); - -- name: InsertUpdateCheck :one INSERT INTO update_checks ( organization_id, product_id, license_id, platform, channel, diff --git a/brainstorming/specs/2026-06-22-client-checkin-data-lifecycle-design.md b/brainstorming/specs/2026-06-22-client-checkin-data-lifecycle-design.md new file mode 100644 index 0000000..c847371 --- /dev/null +++ b/brainstorming/specs/2026-06-22-client-checkin-data-lifecycle-design.md @@ -0,0 +1,142 @@ +# Client check-in data lifecycle design + +## Context + +Client sync currently enqueues one `client_checkins` row containing organization, product, license, activation, version, build, platform, architecture, and OS version. A 90-day cleanup job deletes old rows. The version-adoption module reads those rows twice: latest row per activation powers current distribution and the reporting-device table; latest row per activation/day powers the historical chart. + +No other product or support path queries `client_checkins`. Device self-service uses `devices` and active `activations`, not check-in history. + +## Requirements + +- Keep customer name/email and device hostname/HMAC-protected HWID behavior. +- Persist current device state: activation time, last seen, version, build, platform, architecture, and OS version. +- Preserve the version-adoption summary, current distribution, reporting-device table, and daily trend. +- Count one device at most once per day regardless of check-in frequency. +- Keep long-term trends without stable device identifiers. +- Retain device-linked raw check-ins for 7 days, solely as a bounded diagnostic and aggregation source. +- Keep raw rows free of IP addresses and User-Agent strings. +- Make aggregation retry-safe and preserve cascading deletion. + +## Discovered behavior + +- Producer: `clientsync.Service.Sync` records only successful authorized syncs with a non-empty version. +- Raw writer and cleanup: `diagnostics.Recorder` and `diagnostics.Repository`. +- Current device list and current distribution: `ListLatestClientCheckins`, limited to devices observed during the requested 1–90-day active window. +- Historical chart: `ListDailyLatestClientVersions`, counting distinct activation/day observations by version. +- UI: `website/src/routes/_dash/updates/adoption.tsx` displays both current device state and historical daily counts. +- There is no per-device history view, support query, export, or other consumer. + +## Chosen approach + +Use **Option B: aggregate before bounded raw data expires**. + +This is the smallest change that preserves exact unique-device daily history and the existing 90-day query interface: + +1. An authorized version-bearing sync synchronously performs one database operation that: + - updates current state on the activation; + - inserts the raw check-in. +2. The recorder retains raw rows for 7 days. +3. Once daily, the aggregation worker recomputes aggregate rows for every closed UTC date still present in raw storage, so yesterday is normally available the next day. +4. It replaces aggregate counts for each recomputed date, then deletes raw rows older than 7 days only when those dates were successfully aggregated. The seven-day window is for diagnostics and retries, not an aggregation delay. +5. Current distribution and the reporting-device table query activation/device current state. +6. Historical trends query daily aggregate rows. + +Recomputation uses `INSERT ... ON CONFLICT ... DO UPDATE` plus removal of obsolete dimensions for the same date. Counts are derived from the latest raw row per activation/day, so retries replace counts rather than increment them. Cleanup only runs after successful aggregation. + +### Why not continuous counters + +Incrementing counters cannot distinguish the first observation from the hundredth without retaining a daily device key. That adds another device-linked dataset and more retry logic. Seven-day raw storage already provides the required deduplication source and short-term diagnostic value. + +### Why retain raw rows for seven days + +No current feature needs historical device events. Seven days gives operators a bounded recent diagnostic window and gives the daily aggregation job several retry opportunities. Ninety days has no demonstrated functional value. + +## Data model + +### `activations` + +Add nullable current-state columns: + +- `last_seen_at timestamptz` +- `current_version text` +- `current_build text` +- `platform text` +- `arch text` +- `os_version text` + +Activation time remains `created_at`. Hostname and HMAC-protected HWID remain on `devices`. + +Current state is scoped to an activation because activation is the lifecycle currently shown and deactivated by product behavior. + +### `client_checkins` + +Keep the existing foreign keys to organization, product, license, and activation with cascading deletion. Keep only seven days. Add no IP or User-Agent fields. + +### `daily_version_adoption` + +Add: + +- `date date` +- `organization_id uuid` +- `product_id uuid` +- `version text` +- `device_count bigint` + +Primary key: `(date, organization_id, product_id, version)`. + +Only version is retained as an analytics dimension. Build, platform, architecture, and OS version remain available in current device state but are excluded from long-term aggregates: the current UI does not chart them, and adding dimensions increases cardinality and re-identification risk without demonstrated value. + +This is device-identifier-free aggregated analytics, not anonymous data. The aggregate contains no activation, license, HWID, hostname, or customer identifier. It retains organization/product linkage for tenant-scoped product analytics. + +## Active-device definition + +For current distribution and the reporting-device table, an active device is an undeactivated activation whose `last_seen_at` is within the caller-selected window (`now() - days`). The existing interface remains 1–90 days and defaults to 30. + +For a historical UTC date, a daily active device is an activation that submitted at least one authorized, version-bearing sync on that date. It contributes once, under the version from its final check-in that day. + +## Migration + +One migration will: + +1. add activation current-state columns; +2. create `daily_version_adoption`; +3. backfill current activation state from each activation's newest existing check-in; +4. backfill aggregate rows from the latest check-in per activation/UTC date; +5. leave existing raw rows for the runtime seven-day cleanup to prune after startup. + +The down migration drops the aggregate table and new activation columns. Existing raw rows remain compatible with the old implementation. + +## Module changes + +- Replace the asynchronous `diagnostics.Recorder` write path with one synchronous repository operation that updates activation state and inserts raw history transactionally. Current device state is product data and must not be silently dropped by a full analytics queue. +- Keep the daily aggregation/cleanup worker asynchronous; it is not on the request path. +- Replace raw-history current queries with activation/device queries. +- Replace the daily raw query with an aggregate query. +- Extend recorder cleanup to aggregate closed dates before deletion. +- Set raw retention explicitly to 7 days through configuration, defaulting to 7 and allowing a positive bounded override if operational debugging needs change. +- Preserve the response shape, route, and UI. Update UI wording from “newest check-in” to “current state” where applicable. + +## Failure behavior + +- A failed state/raw write fails the sync request rather than returning success with stale current device state. +- Aggregation remains off the request path and retries from retained raw rows. +- Aggregation failure prevents deletion, so source rows remain for retry. +- Repeated aggregation replaces the same date/version counts and cannot double-count. + +## Validation + +- Repository integration test: repeated check-ins for one activation/day yield one aggregate device; the final version wins. +- Repository integration test: repeated aggregation is idempotent. +- Repository integration test: aggregation failure/absence prevents cleanup; successful aggregation permits cleanup. +- Service test: current distribution uses current activation state and daily trend uses aggregate rows. +- Cleanup-worker test: configured retention is passed to cleanup without affecting the sync request path. +- Migration/schema generation: run sqlc generation/check used by the repository. +- Backend targeted tests: `go test ./internal/features/diagnostics ./internal/features/clientsync` from `backend/`. +- Frontend type/build check using the existing website script. + +## Excluded scope + +- Per-device historical check-in UI or export. +- Aggregate dimensions not consumed by the product. +- IP address or User-Agent collection for client sync. +- Changes to customer identity, hostname, HWID, licensing, or self-service deactivation behavior. diff --git a/brainstorming/specs/2026-06-22-persisted-data-retention-design.md b/brainstorming/specs/2026-06-22-persisted-data-retention-design.md new file mode 100644 index 0000000..5a733da --- /dev/null +++ b/brainstorming/specs/2026-06-22-persisted-data-retention-design.md @@ -0,0 +1,47 @@ +# Persisted data retention design + +## Scope + +Add explicit lifecycle management for temporary/security data and audit history. Preserve licensing semantics, devices, customer identity, orphaned admin accounts, and the client check-in lifecycle. + +## Policies + +| Dataset | Policy | +| --- | --- | +| Self-service tokens | Delete one day after expiry or consumption. | +| Admin sessions | Keep pgxstore's built-in five-minute expired-session cleanup. | +| Organization invites | Delete accepted invites after 30 days and unaccepted invites 30 days after expiry. | +| Admin MFA email codes | Delete one day after expiry or consumption. | +| Audit IP/User-Agent | Set to `NULL` after 90 days. | +| Audit core events | Delete after 180 days. | +| Update checks | Delete after 90 days. Reject zero/unbounded configuration. | +| MCP tokens | Retain until regeneration or organization deletion. | +| Delta jobs | Retain with their release/artifact lifecycle. | +| Licenses/devices/admin users | No automated lifecycle change pending product/legal decisions. | +| Client check-ins | Preserve the separately implemented lifecycle. | + +Audit metadata retention is configurable from 30–180 days, audit event retention from 90–365 days, and update-check retention from 7–365 days. Metadata retention cannot exceed event retention. Fixed grace periods for short-lived records remain code constants because they are part of the credential/invite lifecycle rather than operator policy. + +## Database cleanup module + +Add one retention module with a small interface: start a worker with a query adapter and policies, and close it during application shutdown. It runs once at startup and every 24 hours. Each dataset operation is independent, idempotent, and best-effort; one failure does not block later operations or requests. Every result logs the affected row count and records an OpenTelemetry counter labelled by operation and outcome. + +Move update-check deletion out of `UpdateCheckRecorder`; it continues recording asynchronously but no longer owns retention. Session and client-check-in cleanup remain specialized because sessions are library-managed and check-ins require aggregation before deletion. + +## Email queue expiry + +Use per-message RabbitMQ expiration rather than a blanket queue TTL: + +- admin MFA code: 10 minutes; +- self-service magic link: 15 minutes; +- organization invite: 7 days, matching invite validity; +- license created/replaced: 7 days, allowing recovery from prolonged SMTP/worker outages while bounding raw license-key and customer-email persistence; +- delta-generation events: no email policy; they retain existing behavior. + +Expiration is attached by the backend publisher, keeping queue topology compatible. Expired messages dead-letter through the existing exchange and are not delivered stale. Successful messages are acknowledged and removed; no completed-delivery history is added. + +## Documentation and validation + +Document stored fields, purpose, sensitivity, retention, deletion mechanism, and exclusions in `SETUP.md`. Add config entries to `.env.production.example`. + +Validate configuration bounds, worker independence/idempotent dispatch, email event expiration, generated sqlc output, backend tests, and diagnostics. Database integration tests may require the local Postgres service. diff --git a/website/src/routes/_dash/updates/adoption.tsx b/website/src/routes/_dash/updates/adoption.tsx index 3a9cd4c..54b6071 100644 --- a/website/src/routes/_dash/updates/adoption.tsx +++ b/website/src/routes/_dash/updates/adoption.tsx @@ -334,7 +334,7 @@ function DeviceTable({ data }: { data: VersionAdoptionResponse }) { Reporting devices - Newest check-in per activation during the last {DAYS} days + Current state for devices seen during the last {DAYS} days