diff --git a/.agents/skills/databuddy-internal/SKILL.md b/.agents/skills/databuddy-internal/SKILL.md index 44280eb27a..4f6bd741b0 100644 --- a/.agents/skills/databuddy-internal/SKILL.md +++ b/.agents/skills/databuddy-internal/SKILL.md @@ -37,6 +37,9 @@ Keep additions **minimal**: one bullet, a new `rg` hint, or a routing note—eno - Shared agent integrations should call `@databuddy/ai/agent` (`askDatabuddyAgent` / `streamDatabuddyAgent`) instead of importing internal MCP run/history helpers directly. - First-party ads attribution work should start by preserving UTMs into registration and signup events only; do not add RPC plumbing, conversion destinations, env hooks, tables, workers, or UI until explicitly needed. - Insights generation logic belongs in `apps/insights` and should reuse `@databuddy/ai`; `apps/api` should only read insight data or queue runs, not own prompts, model calls, tool loops, validation, or persistence orchestration. +- Automated insight digests have one organization-wide schedule (`off`, `daily`, or `weekly`) and one organization-wide delivery set; website selection is only for manual runs. Do not reintroduce per-website overrides, hourly/custom cadence, or cron input. +- Insight run items are execution metadata, not rendered insight content; previews should use run status/counts or query real insights, never infer titles or bodies from run items. +- Insight Slack delivery must resolve each channel binding to its active same-organization integration; never choose an arbitrary organization bot token. - Agent ClickHouse SQL must use the canonical analytics.events schema: `client_id`, `time`, `path`, `event_name`, and pageviews as `event_name = 'screen_view'`; never `website_id`, `created_at`, `page_path`, `event_type`, or `pageview`. - Slack agent evals live in `packages/evals`: use `bun run eval --surface slack` for the whole Slack surface. `--tag slack` is only a tiny smoke subset, and `cost_fallback` in agent telemetry is pricing-catalog fallback, not proof the model request fell back. - Slack agent expected stops such as exhausted Databunny credits should throw `DatabuddyAgentUserError` from `@databuddy/ai/agent/errors`; Slack surfaces those messages directly and reserves the generic reconnect copy for real infrastructure failures. @@ -53,6 +56,7 @@ Keep additions **minimal**: one bullet, a new `rg` hint, or a routing note—eno - Public status pages render from `apps/status`; `apps/dashboard` owns status-page management/config UI only. When cleaning public status UX, update shared `@databuddy/ui/uptime` pieces or `apps/status` wrappers instead of redesigning dashboard-only route remnants. - `packages/db`: Drizzle Postgres schema, client, and ClickHouse helpers - `packages/rpc`: shared oRPC router, procedures, auth-aware server context +- `packages/rpc` must declare `drizzle-orm: "catalog:"` before importing `drizzle-orm/*` helpers such as `drizzle-orm/zod`; otherwise TypeScript can resolve a different Drizzle instance than `@databuddy/db` and reject table-derived schemas. - `packages/auth`: Better Auth setup, permissions, organization access - `packages/env`: per-app env schemas - `packages/shared`: shared types, flags, analytics schemas, utilities @@ -109,7 +113,7 @@ Read [codebase-map.md](./references/codebase-map.md) when you need deeper routin - Funnel rows keep the action menu outside the main toggle button; put row padding on the sibling `Button`, not only on `List.Row`, so the visible row surface is clickable without nesting buttons. - Demo website navigation must be public-safe and route-backed; hide sensitive, configuration-heavy, or unavailable website features such as Agent, Feature Flags, Revenue, Users, Realtime, Anomalies, and website Settings instead of inheriting the full website nav. Goals and Funnels may be public demo surfaces, but keep them read-only. - Dashboard definitions for feature flags and target groups are admin surfaces; do not expose even sanitized rows to demo-tier/public website access. -- Insights merged feed (`use-insights-feed`) collapses history + AI by `insightSignalDedupeKey` in `apps/dashboard/lib/insight-signal-key.ts` so the list is one row per signal (latest wins). +- Insights feed (`use-insights-feed`) collapses persisted history by `insightSignalDedupeKey` in `apps/dashboard/lib/insight-signal-key.ts` so the list is one row per signal (latest wins); reads must not invoke AI generation. - Insights page (`app/(main)/insights`) should stay focused on the brief + signal queue; do not add generic global analytics KPI cards or top pages/referrers/countries tables there. - Theme: `apps/dashboard/app/globals.css`. **`--border` is intentionally subtle**; do not crank it darker for “contrast” unless **iza** asks—prefer text tokens or layout for readability. - Website analytics filters are two-way synced between Jotai and the `filters` URL param in `app/(main)/websites/[id]/layout.tsx`; guard URL-driven atom writes from echoing stale atom state back into `nuqs`, or adding a filter can lock the page during form submit. @@ -139,6 +143,7 @@ Read [codebase-map.md](./references/codebase-map.md) when you need deeper routin ## Billing (Autumn) +- Transactional billing email identity has three separate concepts: Autumn customer/billing owner, organization, and actual `to` recipient. Only personalize from the actual recipient record; if it is unavailable, omit the greeting rather than using the owner name. Keep `agent_credits` as an internal feature ID, but describe it to customers as Databunny usage or an AI analysis allowance and explain what it enables. - `autumn-js` v1.2.2+ — import `autumnHandler` from `autumn-js/fetch` (NOT `autumn-js/elysia`, that export was removed in v1.0) - For Elysia, mount with `.mount(autumnHandler(...))` — NOT `.use()` - `identify` callback receives `(request: Request)` directly, not `({ request })` @@ -168,6 +173,7 @@ Read [codebase-map.md](./references/codebase-map.md) when you need deeper routin - Published SDK logic: `packages/sdk/src` - Browser tracker bundle: `packages/tracker/src` +- When a retryable batch failure restores events to an in-memory queue, it must also restore an automatic retry timer with capped backoff; requeueing alone silently stalls delivery. - Public SDK/tracker visitor ID privacy is only `anonymizeVisitorIds` (`true`/omitted = anonymized, `false` = raw IDs, `"auto"` = raw only in Databuddy's conservative country allowlist). - Keep visitor ID privacy internals small and direct; avoid exported helper stacks or storage/hashing vocabulary for this option. - If the user reports missing analytics events, inspect both the producer side and `apps/basket` diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 14f1024f98..873e7d5d74 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -43,6 +43,8 @@ jobs: name: Check Types runs-on: blacksmith-4vcpu-ubuntu-2404 timeout-minutes: 15 + env: + CLICKHOUSE_READONLY_URL: ${{ secrets.CLICKHOUSE_READONLY_URL }} steps: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 @@ -56,6 +58,11 @@ jobs: - run: bun install --frozen-lockfile --ignore-scripts - run: bun run --cwd apps/docs postinstall - run: bun run check-types + - name: ClickHouse types match DDL + run: bun run --cwd packages/db ch:check + - name: ClickHouse schema matches cluster + if: env.CLICKHOUSE_READONLY_URL != '' + run: bun run --cwd packages/db ch:verify test: name: Test diff --git a/apps/api/src/http/errors.test.ts b/apps/api/src/http/errors.test.ts index 7f7d0153cf..eaf20a710f 100644 --- a/apps/api/src/http/errors.test.ts +++ b/apps/api/src/http/errors.test.ts @@ -1,4 +1,5 @@ import { createError } from "evlog"; +import { t, ValidationError } from "elysia"; import { afterEach, describe, expect, it } from "vitest"; import { handleAppError } from "./errors"; @@ -16,6 +17,7 @@ describe("handleAppError", () => { it("masks structured 5xx details in production", async () => { process.env.NODE_ENV = "production"; const response = handleAppError({ + requestId: "req_test_5xx", error: createError({ code: "api.SECRET_FAILURE", message: "Database password leaked into error", @@ -27,16 +29,19 @@ describe("handleAppError", () => { }); expect(response.status).toBe(500); - expect(await readPayload(response)).toEqual({ - success: false, - error: "An internal server error occurred", - code: "api.SECRET_FAILURE", - }); + expect(await readPayload(response)).toEqual({ + success: false, + error: "An internal server error occurred", + code: "api.SECRET_FAILURE", + requestId: "req_test_5xx", + }); + expect(response.headers.get("X-Request-ID")).toBe("req_test_5xx"); }); it("keeps structured 4xx details visible in production", async () => { process.env.NODE_ENV = "production"; const response = handleAppError({ + requestId: "req_test_4xx", error: createError({ code: "api.BAD_INPUT", message: "Invalid filter", @@ -47,12 +52,38 @@ describe("handleAppError", () => { }); expect(response.status).toBe(400); - expect(await readPayload(response)).toEqual({ + expect(await readPayload(response)).toEqual({ success: false, error: "Invalid filter", code: "api.BAD_INPUT", why: "The filter operator is unsupported.", - fix: "Use one of the documented operators.", + fix: "Use one of the documented operators.", + requestId: "req_test_4xx", + }); + }); + + it("returns safe field details for request validation errors", async () => { + process.env.NODE_ENV = "production"; + const response = handleAppError({ + code: "VALIDATION", + requestId: "req_test_validation", + error: new ValidationError( + "body", + t.Object({ name: t.String({ minLength: 1 }) }), + {} + ), + }); + const payload = await readPayload(response); + + expect(response.status).toBe(422); + expect(payload).toMatchObject({ + success: false, + error: "Invalid request", + code: "VALIDATION", + requestId: "req_test_validation", }); + expect(payload.details).toEqual([ + expect.objectContaining({ field: "body.name" }), + ]); }); }); diff --git a/apps/api/src/http/errors.ts b/apps/api/src/http/errors.ts index f8e04df75f..94759eef2d 100644 --- a/apps/api/src/http/errors.ts +++ b/apps/api/src/http/errors.ts @@ -1,16 +1,20 @@ import { config } from "@databuddy/env/app"; +import { ValidationError } from "elysia"; import { EvlogError, parseError } from "evlog"; +import { getRequestId } from "./request-id"; interface AppErrorContext { code?: string | number; error: unknown; + request?: Request; + requestId?: string; } const HTTP_STATUS_BY_ERROR_CODE: Record = { AUTH_REQUIRED: 401, BAD_REQUEST: 400, CONFLICT: 409, - FEATURE_UNAVAILABLE: 403, + FEATURE_UNAVAILABLE: 402, FORBIDDEN: 403, INTERNAL_SERVER_ERROR: 500, INVALID_COOKIE_SIGNATURE: 400, @@ -25,8 +29,15 @@ const HTTP_STATUS_BY_ERROR_CODE: Record = { }; const PROTECTED_RESOURCE_METADATA_URL = `${config.urls.api}/.well-known/oauth-protected-resource`; +const LEADING_SLASH_PATTERN = /^\//; -export function handleAppError({ error, code }: AppErrorContext) { +export function handleAppError({ + error, + code, + request, + requestId, +}: AppErrorContext) { + const responseRequestId = requestId ?? getRequestId(request); const parsed = parseError(error); const statusCode = getStatusCode({ code, @@ -48,8 +59,10 @@ export function handleAppError({ error, code }: AppErrorContext) { isClientError, statusCode, }); + const validationDetails = getValidationDetails(error); const headers: Record = { "Content-Type": "application/json", + "X-Request-ID": responseRequestId, }; if (statusCode === 401) { headers["WWW-Authenticate"] = @@ -61,16 +74,55 @@ export function handleAppError({ error, code }: AppErrorContext) { success: false, error: safeClientError, code: errorCode, + requestId: responseRequestId, ...(hasValue(parsed.why) && exposeStructured ? { why: parsed.why } : {}), ...(hasValue(parsed.fix) && exposeStructured ? { fix: parsed.fix } : {}), ...(hasValue(parsed.link) && exposeStructured ? { link: parsed.link } : {}), + ...(validationDetails.length > 0 ? { details: validationDetails } : {}), }), { status: statusCode, headers } ); } +interface ValidationDetail { + field: string; + message: string; +} + +function getValidationDetails(error: unknown): ValidationDetail[] { + if (!(error instanceof ValidationError) || error.type === "response") { + return []; + } + + const details: ValidationDetail[] = []; + const seenFields = new Set(); + for (const issue of error.all) { + const path = issue.path + .replace(LEADING_SLASH_PATTERN, "") + .split("/") + .filter(Boolean) + .join("."); + const field = path ? `${error.type}.${path}` : error.type; + if (seenFields.has(field)) { + continue; + } + seenFields.add(field); + details.push({ + field, + message: + typeof issue.summary === "string" && issue.summary + ? issue.summary + : issue.message, + }); + if (details.length === 20) { + break; + } + } + return details; +} + function getErrorCode({ explicitCode, parsedCode, diff --git a/apps/api/src/http/request-id.test.ts b/apps/api/src/http/request-id.test.ts new file mode 100644 index 0000000000..d5c167a495 --- /dev/null +++ b/apps/api/src/http/request-id.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from "vitest"; +import { getRequestId } from "./request-id"; + +describe("getRequestId", () => { + it("returns a stable generated ID for the same request", () => { + const request = new Request("https://api.example.com/test"); + const requestId = getRequestId(request); + + expect(requestId).toMatch(/^req_[a-f0-9]{16}$/); + expect(getRequestId(request)).toBe(requestId); + }); + + it("preserves a safe caller-supplied ID", () => { + const request = new Request("https://api.example.com/test", { + headers: { "x-request-id": "trace_partner-123" }, + }); + + expect(getRequestId(request)).toBe("trace_partner-123"); + }); + + it("replaces unsafe caller-supplied IDs", () => { + const request = new Request("https://api.example.com/test", { + headers: { "x-request-id": "not safe for logs" }, + }); + + expect(getRequestId(request)).toMatch(/^req_[a-f0-9]{16}$/); + }); +}); diff --git a/apps/api/src/http/request-id.ts b/apps/api/src/http/request-id.ts new file mode 100644 index 0000000000..403dd17537 --- /dev/null +++ b/apps/api/src/http/request-id.ts @@ -0,0 +1,25 @@ +const REQUEST_ID_PATTERN = /^[A-Za-z0-9_-]{1,128}$/; +const requestIds = new WeakMap(); + +function createRequestId(): string { + return `req_${crypto.randomUUID().replaceAll("-", "").slice(0, 16)}`; +} + +export function getRequestId(request?: Request): string { + if (!request) { + return createRequestId(); + } + + const existing = requestIds.get(request); + if (existing) { + return existing; + } + + const supplied = request.headers.get("x-request-id")?.trim(); + const requestId = + supplied && REQUEST_ID_PATTERN.test(supplied) + ? supplied + : createRequestId(); + requestIds.set(request, requestId); + return requestId; +} diff --git a/apps/api/src/index.ts b/apps/api/src/index.ts index 066b109697..444f58eed7 100644 --- a/apps/api/src/index.ts +++ b/apps/api/src/index.ts @@ -10,6 +10,7 @@ import { registerProcessErrorHandlers } from "@/bootstrap/process-errors"; import { registerShutdownHooks, warmPostgresPool } from "@/bootstrap/shutdown"; import { isAllowedApiOrigin } from "@/http/cors"; import { handleAppError } from "@/http/errors"; +import { getRequestId } from "@/http/request-id"; import { AUTUMN_API_PREFIX } from "@/lib/autumn-mount"; import { enrichApiWideEvent } from "@/lib/evlog-api"; import { enrichRequestAuthWideEvent } from "@/middleware/auth-wide-event"; @@ -81,10 +82,20 @@ const app = new Elysia({ precompile: true }) enrich: enrichApiWideEvent, }) ) + .onBeforeHandle(({ request, set }) => { + set.headers["X-Request-ID"] = getRequestId(request); + }) .onBeforeHandle(({ request }) => enrichRequestAuthWideEvent(request)) .use( cors({ credentials: true, + exposeHeaders: [ + "X-Request-ID", + "Retry-After", + "X-RateLimit-Limit", + "X-RateLimit-Remaining", + "X-RateLimit-Reset", + ], origin: isAllowedApiOrigin, }) ) diff --git a/apps/api/src/integration/link-handlers.test.ts b/apps/api/src/integration/link-handlers.test.ts index 6bdcbc70fc..82abc8a22b 100644 --- a/apps/api/src/integration/link-handlers.test.ts +++ b/apps/api/src/integration/link-handlers.test.ts @@ -263,3 +263,88 @@ describe("links.list", () => { expect(result[0].name).toBe("Org A Link"); }); }); + +describe("links.paginated", () => { + iit("paginates results and reports hasMore", async () => { + const user = await signUp(); + const org = await insertOrganization(); + await addToOrganization(user.id, org.id, "member"); + + for (let i = 0; i < 3; i++) { + await call(appRouter.links.create, userContext(user, org.id))({ + name: `Link ${i}`, + targetUrl: `https://link-${i}.example.com`, + organizationId: org.id, + }); + } + + const firstPage = await call( + appRouter.links.paginated, + userContext(user, org.id), + )({ organizationId: org.id, limit: 2, offset: 0 }); + + expect(firstPage.items).toHaveLength(2); + expect(firstPage.hasMore).toBe(true); + + const secondPage = await call( + appRouter.links.paginated, + userContext(user, org.id), + )({ organizationId: org.id, limit: 2, offset: 2 }); + + expect(secondPage.items).toHaveLength(1); + expect(secondPage.hasMore).toBe(false); + }); + + iit("filters by search term", async () => { + const user = await signUp(); + const org = await insertOrganization(); + await addToOrganization(user.id, org.id, "member"); + + await call(appRouter.links.create, userContext(user, org.id))({ + name: "Summer Campaign", + targetUrl: "https://summer.example.com", + organizationId: org.id, + }); + await call(appRouter.links.create, userContext(user, org.id))({ + name: "Winter Promo", + targetUrl: "https://winter.example.com", + organizationId: org.id, + }); + + const result = await call( + appRouter.links.paginated, + userContext(user, org.id), + )({ organizationId: org.id, search: "summer" }); + + expect(result.items).toHaveLength(1); + expect(result.items[0].name).toBe("Summer Campaign"); + }); + + iit("does not leak links from other orgs", async () => { + const userA = await signUp(); + const userB = await signUp(); + const orgA = await insertOrganization(); + const orgB = await insertOrganization(); + await addToOrganization(userA.id, orgA.id, "member"); + await addToOrganization(userB.id, orgB.id, "member"); + + await call(appRouter.links.create, userContext(userA, orgA.id))({ + name: "Org A Link", + targetUrl: "https://a.example.com", + organizationId: orgA.id, + }); + await call(appRouter.links.create, userContext(userB, orgB.id))({ + name: "Org B Link", + targetUrl: "https://b.example.com", + organizationId: orgB.id, + }); + + const result = await call( + appRouter.links.paginated, + userContext(userA, orgA.id), + )({ organizationId: orgA.id }); + + expect(result.items).toHaveLength(1); + expect(result.items[0].name).toBe("Org A Link"); + }); +}); diff --git a/apps/api/src/integration/profile-handlers.test.ts b/apps/api/src/integration/profile-handlers.test.ts index 0cbe3ca745..4cd8824871 100644 --- a/apps/api/src/integration/profile-handlers.test.ts +++ b/apps/api/src/integration/profile-handlers.test.ts @@ -8,6 +8,7 @@ import { import { eq } from "@databuddy/db"; import { appRouter, type Context } from "@databuddy/rpc"; import { + getTraitDistribution, resolveTraitSegment, splitTraits, upsertProfile, @@ -251,6 +252,33 @@ describe("upsertProfile trait history", () => { }); }); +describe("profiles.findByEmail", () => { + iit("finds a profile by exact email via the lookup hash", async () => { + const user = await signUp(); + const org = await insertOrganization(); + await addToOrganization(user.id, org.id, "member"); + const website = await insertWebsite({ organizationId: org.id }); + await upsertProfile( + website.id, + "user_1", + splitTraits({ email: "Jo@Acme.com", name: "Jo" }) + ); + const ctx = userContext(user, org.id); + + const found = await call(appRouter.profiles.findByEmail, ctx)({ + websiteId: website.id, + email: "jo@acme.com", + }); + expect(found).toEqual({ profileId: "user_1" }); + + const missing = await call(appRouter.profiles.findByEmail, ctx)({ + websiteId: website.id, + email: "nobody@acme.com", + }); + expect(missing).toBeNull(); + }); +}); + describe("profiles.traitKeys / traitValues", () => { iit("lists distinct keys and values for the website only", async () => { const user = await signUp(); @@ -292,6 +320,26 @@ describe("profiles.traitKeys / traitValues", () => { }); }); +describe("getTraitDistribution", () => { + iit("ranks values per key with profile counts", async () => { + const org = await insertOrganization(); + const website = await insertWebsite({ organizationId: org.id }); + await seedProfile(website.id, "u1", { traits: { plan: "pro" } }); + await seedProfile(website.id, "u2", { traits: { plan: "pro" } }); + await seedProfile(website.id, "u3", { + traits: { plan: "free", beta: true }, + }); + + const distribution = await getTraitDistribution(website.id); + expect(distribution.identifiedProfiles).toBe(3); + expect(distribution.traits).toEqual([ + { key: "beta", value: "true", profiles: 1 }, + { key: "plan", value: "pro", profiles: 2 }, + { key: "plan", value: "free", profiles: 1 }, + ]); + }); +}); + describe("resolveTraitSegment", () => { iit("resolves profile ids by trait predicate scoped to the website", async () => { const org = await insertOrganization(); diff --git a/apps/api/src/lib/evlog-api.ts b/apps/api/src/lib/evlog-api.ts index 43bd59ea8c..12af502421 100644 --- a/apps/api/src/lib/evlog-api.ts +++ b/apps/api/src/lib/evlog-api.ts @@ -30,6 +30,8 @@ const devFsLogsDir = join( const useLocalEvlogFiles = process.env.NODE_ENV === "development" || readBooleanEnv("API_EVLOG_FS"); +const drainToAxiom = process.env.NODE_ENV !== "development"; + const devFsDrain = useLocalEvlogFiles ? createFsDrain({ dir: devFsLogsDir, pretty: false }) : null; @@ -93,7 +95,9 @@ export async function apiLoggerDrain(ctx: DrainContext): Promise { if (devFsDrain) { await devFsDrain(ctx); } - batchedAxiomDrain(ctx); + if (drainToAxiom) { + batchedAxiomDrain(ctx); + } batchedSuperlogDrain?.(ctx); } diff --git a/apps/api/src/routes/agent.ts b/apps/api/src/routes/agent.ts index 3c1b444377..7ab75adc23 100644 --- a/apps/api/src/routes/agent.ts +++ b/apps/api/src/routes/agent.ts @@ -636,7 +636,7 @@ export const agent = new Elysia({ prefix: "/v1/agent" }) return jsonError( 400, "WORKSPACE_REQUIRED", - "No active workspace. Select an organization and try again." + "No active organization. Select an organization and try again." ); } @@ -681,7 +681,7 @@ export const agent = new Elysia({ prefix: "/v1/agent" }) return jsonError( 403, "ACCESS_DENIED", - "No accessible websites in this workspace" + "No accessible websites in this organization" ); } @@ -832,7 +832,7 @@ export const agent = new Elysia({ prefix: "/v1/agent" }) return jsonError( 402, "OUT_OF_CREDITS", - "You're out of Databunny credits this month. Upgrade or wait for the monthly reset." + "You've used your Databunny allowance for this month. Add more usage, upgrade, or wait for the monthly reset." ); } diff --git a/apps/api/src/routes/integrations/index.ts b/apps/api/src/routes/integrations/index.ts index b8d189aec9..5e342f6233 100644 --- a/apps/api/src/routes/integrations/index.ts +++ b/apps/api/src/routes/integrations/index.ts @@ -447,7 +447,12 @@ export const integrations = new Elysia({ prefix: "/v1/integrations" }) slack_oauth: "callback", provider_error: query.error, }); - return integrationsRedirect("error", "Slack authorization failed"); + return integrationsRedirect( + "error", + query.error === "access_denied" + ? "Slack wasn't connected because authorization was canceled" + : "Slack authorization failed" + ); } try { if (!(query.code && query.state)) { diff --git a/apps/api/src/routes/public/flags.ts b/apps/api/src/routes/public/flags.ts index 2303c1275f..c4f3d701c5 100644 --- a/apps/api/src/routes/public/flags.ts +++ b/apps/api/src/routes/public/flags.ts @@ -108,6 +108,17 @@ const bulkFlagQuerySchema = t.Object({ environment: t.Optional(t.String()), }); +const bulkFlagBodySchema = t.Object({ + clientId: t.String(), + keys: t.Optional(t.Array(t.String())), + userId: t.Optional(t.String()), + email: t.Optional(t.String()), + organizationId: t.Optional(t.String()), + teamId: t.Optional(t.String()), + properties: t.Optional(t.Record(t.String(), t.Any())), + environment: t.Optional(t.String()), +}); + const getCachedFlag = cacheable( async (key: string, clientId: string, environment?: string) => { const flag = await db.query.flags.findFirst({ @@ -731,6 +742,100 @@ function buildFlagChangeSnapshot(flag: typeof flags.$inferSelect) { }; } +interface BulkFlagInput extends UserContext { + clientId: string; + environment?: string; + keys?: string[]; +} + +async function evaluateBulkFlags( + input: BulkFlagInput, + set: ElysiaSet, + request: Request +) { + if (!(await enforcePublicFlagRateLimit(request, input.clientId, set))) { + return { flags: {}, count: 0, reason: "RATE_LIMITED" }; + } + + mergeWideEvent({ + flag_bulk: true, + flag_client_id: input.clientId || "", + flag_has_user_id: Boolean(input.userId), + flag_has_email: Boolean(input.email), + flag_environment: input.environment || "", + }); + + try { + if (!input.clientId) { + mergeWideEvent({ flag_error: "missing_client_id" }); + set.status = 400; + return { + flags: {}, + count: 0, + error: "Missing required clientId parameter", + }; + } + + const context: UserContext = { + userId: input.userId, + email: input.email, + organizationId: input.organizationId, + teamId: input.teamId, + properties: input.properties, + }; + const requestedKeys = input.keys + ? new Set(input.keys.map((key) => key.trim()).filter(Boolean)) + : null; + + const clientFlags = await fromMemory( + `fc:${input.clientId}:${input.environment || ""}`, + () => getCachedFlagsForClient(input.clientId, input.environment) + ); + let allFlags = clientFlags; + + if (context.userId) { + const userId = context.userId; + const userFlags = await fromMemory( + `fu:${userId}:${input.clientId}:${input.environment || ""}`, + () => getCachedFlagsForUser(userId, input.clientId, input.environment) + ); + if (userFlags.length > 0) { + const clientKeys = new Set(clientFlags.map((flag) => flag.key)); + const uniqueUserFlags = userFlags.filter( + (flag) => !clientKeys.has(flag.key) + ); + allFlags = [...clientFlags, ...uniqueUserFlags]; + } + } + + const flagsToEvaluate = requestedKeys + ? allFlags.filter((flag) => requestedKeys.has(flag.key)) + : allFlags; + const results: Record = {}; + for (const flag of flagsToEvaluate) { + results[flag.key] = dependenciesSatisfiedFromList(flag, allFlags) + ? evaluateFlag(flag, context) + : dependencyFailure(); + } + + const count = Object.keys(results).length; + mergeWideEvent({ + flag_total_flags: allFlags.length, + flag_evaluated: flagsToEvaluate.length, + flag_count: count, + }); + return { flags: results, count }; + } catch (error) { + mergeWideEvent({ flag_error: true }); + useLogger().error( + error instanceof Error ? error : new Error(String(error)), + { flags: { bulk: true, clientId: input.clientId } } + ); + set.status = 500; + return { flags: {}, count: 0, error: "Bulk evaluation failed" }; + } +} + const variantBodySchema = t.Object({ key: t.String({ minLength: 1 }), type: t.Union([t.Literal("string"), t.Literal("number"), t.Literal("json")]), @@ -742,10 +847,20 @@ const variantBodySchema = t.Object({ export const flagsRoute = new Elysia({ prefix: "/v1/flags" }) .onAfterHandle(({ set, request }) => { if (!set.status || set.status === 200) { - const pathname = new URL(request.url).pathname; - set.headers["cache-control"] = PUBLIC_CACHE_PATH_RE.test(pathname) - ? FLAG_CACHE_CONTROL - : "private, no-store"; + const url = new URL(request.url); + const hasTargetingContext = [ + "userId", + "email", + "organizationId", + "teamId", + "properties", + ].some((key) => url.searchParams.has(key)); + set.headers["cache-control"] = + request.method === "GET" && + !hasTargetingContext && + PUBLIC_CACHE_PATH_RE.test(url.pathname) + ? FLAG_CACHE_CONTROL + : "private, no-store"; } set.headers.vary = "Origin"; }) @@ -848,100 +963,31 @@ export const flagsRoute = new Elysia({ prefix: "/v1/flags" }) .get( "/bulk", - async function bulkEvaluateFlags({ query, set, request }) { - if (!(await enforcePublicFlagRateLimit(request, query.clientId, set))) { - return { flags: [], reason: "RATE_LIMITED" }; - } - - mergeWideEvent({ - flag_bulk: true, - flag_client_id: query.clientId || "", - flag_has_user_id: Boolean(query.userId), - flag_has_email: Boolean(query.email), - flag_environment: query.environment || "", - }); - - try { - if (!query.clientId) { - mergeWideEvent({ flag_error: "missing_client_id" }); - set.status = 400; - return { - flags: {}, - count: 0, - error: "Missing required clientId parameter", - }; - } - - const context: UserContext = { + function bulkEvaluateFlags({ query, set, request }) { + return evaluateBulkFlags( + { + clientId: query.clientId, + keys: query.keys?.split(","), userId: query.userId, email: query.email, organizationId: query.organizationId, teamId: query.teamId, properties: parseProperties(query.properties), - }; - - const requestedKeys = query.keys - ? new Set( - query.keys - .split(",") - .map((k) => k.trim()) - .filter(Boolean) - ) - : null; - - const clientFlags = await fromMemory( - `fc:${query.clientId}:${query.environment || ""}`, - () => getCachedFlagsForClient(query.clientId, query.environment) - ); - - let allFlags = clientFlags; - - if (context.userId) { - const uid = context.userId; - const userFlags = await fromMemory( - `fu:${uid}:${query.clientId}:${query.environment || ""}`, - () => getCachedFlagsForUser(uid, query.clientId, query.environment) - ); - if (userFlags.length > 0) { - const clientKeys = new Set(clientFlags.map((f) => f.key)); - const uniqueUserFlags = userFlags.filter( - (f) => !clientKeys.has(f.key) - ); - allFlags = [...clientFlags, ...uniqueUserFlags]; - } - } - - const flagsToEvaluate = requestedKeys - ? allFlags.filter((f) => requestedKeys.has(f.key)) - : allFlags; - - const results: Record = {}; - for (const flag of flagsToEvaluate) { - results[flag.key] = dependenciesSatisfiedFromList(flag, allFlags) - ? evaluateFlag(flag, context) - : dependencyFailure(); - } - - const count = Object.keys(results).length; - mergeWideEvent({ - flag_total_flags: allFlags.length, - flag_evaluated: flagsToEvaluate.length, - flag_count: count, - }); - - return { flags: results, count }; - } catch (error) { - mergeWideEvent({ flag_error: true }); - useLogger().error( - error instanceof Error ? error : new Error(String(error)), - { flags: { bulk: true, clientId: query.clientId } } - ); - set.status = 500; - return { flags: {}, count: 0, error: "Bulk evaluation failed" }; - } + environment: query.environment, + }, + set, + request + ); }, { query: bulkFlagQuerySchema } ) + .post( + "/bulk", + function bulkEvaluateFlagsPost({ body, set, request }) { + return evaluateBulkFlags(body, set, request); + }, + { body: bulkFlagBodySchema } + ) .get( "/definitions", diff --git a/apps/api/src/routes/public/index.ts b/apps/api/src/routes/public/index.ts index 921921759a..6b4eee33e6 100644 --- a/apps/api/src/routes/public/index.ts +++ b/apps/api/src/routes/public/index.ts @@ -22,13 +22,14 @@ export const publicApi = new Elysia({ prefix: "/public" }) .use( cors({ credentials: false, + exposeHeaders: ["X-Request-ID", "Retry-After"], origin: true, }) ) .options("*", () => new Response(null, { status: 204 })) .use(agentTelemetryRoute) .use(flagsRoute) - .onError(function handlePublicError({ error, code }) { + .onError(function handlePublicError({ error, code, request }) { const isNotFound = code === "NOT_FOUND"; mergeWideEvent({ public_api: true, @@ -38,5 +39,5 @@ export const publicApi = new Elysia({ prefix: "/public" }) captureError(error, { public_api: true }); } - return handleAppError({ code, error }); + return handleAppError({ code, error, request }); }); diff --git a/apps/api/src/routes/query.ts b/apps/api/src/routes/query.ts index fbd2107b7d..cdf45ac42e 100644 --- a/apps/api/src/routes/query.ts +++ b/apps/api/src/routes/query.ts @@ -19,7 +19,7 @@ import { } from "@databuddy/services/identity"; import { validateTimezone } from "@databuddy/validation"; import { readBooleanEnv } from "@databuddy/env/boolean"; -import { ratelimit } from "@databuddy/redis/rate-limit"; +import { getRateLimitHeaders, ratelimit } from "@databuddy/redis/rate-limit"; import { getBillingOwner } from "@databuddy/rpc/billing"; import { getOrganizationOwnerId } from "@databuddy/rpc/organization"; import { @@ -59,6 +59,7 @@ import { DynamicQueryRequestSchema, type DynamicQueryRequestType, } from "../schemas/query-schemas"; +import { getRequestId } from "../http/request-id"; const parsedPerWebsiteQueryConcurrency = Number( process.env.PER_WEBSITE_QUERY_CONCURRENCY ?? 8 @@ -311,10 +312,6 @@ function validatePaginationFields( return errors; } -function generateRequestId(): string { - return `req_${crypto.randomUUID().replace(/-/g, "").slice(0, 16)}`; -} - interface AuthContext { // Session active org; used when query omits organization_id. activeOrganizationId: string | null; @@ -351,6 +348,7 @@ function createAuthFailedResponse(requestId: string): Response { status: 401, headers: { "Content-Type": "application/json", + "X-Request-ID": requestId, "WWW-Authenticate": `Bearer resource_metadata="${PROTECTED_RESOURCE_METADATA_URL}"`, }, } @@ -391,20 +389,24 @@ async function enforceQueryRateLimit( if (rl.success) { return null; } + const retryAfter = Math.max(1, Math.ceil((rl.reset - Date.now()) / 1000)); return new Response( JSON.stringify({ success: false, error: "Rate limit exceeded", code: "RATE_LIMITED", requestId, + limit: rl.limit, + remaining: rl.remaining, + reset: rl.reset, + retryAfter, }), { status: 429, headers: { "Content-Type": "application/json", - "X-RateLimit-Limit": String(rl.limit), - "X-RateLimit-Remaining": String(rl.remaining), - "X-RateLimit-Reset": String(rl.reset), + "X-Request-ID": requestId, + ...getRateLimitHeaders(rl), }, } ); @@ -420,6 +422,9 @@ function createErrorResponse( const headers: Record = { "Content-Type": "application/json", }; + if (requestId) { + headers["X-Request-ID"] = requestId; + } if (status === 401) { headers["WWW-Authenticate"] = `Bearer resource_metadata="${PROTECTED_RESOURCE_METADATA_URL}"`; @@ -1216,9 +1221,9 @@ export const query = new Elysia({ prefix: "/v1/query" }) }; }) - .get("/websites", ({ auth: ctx }) => + .get("/websites", ({ auth: ctx, request }) => (async () => { - const requestId = generateRequestId(); + const requestId = getRequestId(request); if (!ctx.isAuthenticated) { return createAuthFailedResponse(requestId); } @@ -1232,29 +1237,38 @@ export const query = new Elysia({ prefix: "/v1/query" }) })() ) - .get("/types", ({ query: params }: { query: { include_meta?: string } }) => { - const requestId = generateRequestId(); - const includeMeta = params.include_meta === "true"; - const configs = Object.fromEntries( - Object.entries(QueryBuilders).map(([key, cfg]) => [ - key, - { - allowedFilters: allowedFilterFields(cfg), - customizable: cfg.customizable, - defaultLimit: cfg.limit, - publicAccess: cfg.publicAccess === true, - ...(includeMeta && { meta: cfg.meta }), - }, - ]) - ); - return { - success: true, - requestId, - types: Object.keys(QueryBuilders), - configs, - presets: Object.keys(DatePresets), - }; - }) + .get( + "/types", + ({ + query: params, + request, + }: { + query: { include_meta?: string }; + request: Request; + }) => { + const requestId = getRequestId(request); + const includeMeta = params.include_meta === "true"; + const configs = Object.fromEntries( + Object.entries(QueryBuilders).map(([key, cfg]) => [ + key, + { + allowedFilters: allowedFilterFields(cfg), + customizable: cfg.customizable, + defaultLimit: cfg.limit, + publicAccess: cfg.publicAccess === true, + ...(includeMeta && { meta: cfg.meta }), + }, + ]) + ); + return { + success: true, + requestId, + types: Object.keys(QueryBuilders), + configs, + presets: Object.keys(DatePresets), + }; + } + ) .post( "/compile", @@ -1269,7 +1283,7 @@ export const query = new Elysia({ prefix: "/v1/query" }) auth: AuthContext; request: Request; }) => { - const requestId = generateRequestId(); + const requestId = getRequestId(request); const rateLimited = await enforceQueryRateLimit( ctx, "compile", @@ -1339,7 +1353,7 @@ export const query = new Elysia({ prefix: "/v1/query" }) request: Request; }) => (async () => { - const requestId = generateRequestId(); + const requestId = getRequestId(request); const timezone = validateTimezone(q.timezone) || "UTC"; const rateLimited = await enforceQueryRateLimit( ctx, @@ -1530,7 +1544,7 @@ export const query = new Elysia({ prefix: "/v1/query" }) request: Request; }) => (async () => { - const requestId = generateRequestId(); + const requestId = getRequestId(request); const rateLimited = await enforceQueryRateLimit( ctx, "custom", diff --git a/apps/api/src/routes/webhooks/autumn.test.ts b/apps/api/src/routes/webhooks/autumn.test.ts index c378d1324c..21e6458057 100644 --- a/apps/api/src/routes/webhooks/autumn.test.ts +++ b/apps/api/src/routes/webhooks/autumn.test.ts @@ -1,6 +1,16 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; const state = vi.hoisted(() => ({ + check: vi.fn(async () => ({ + allowed: true, + balance: { + granted: 10_000, + nextResetAt: Date.UTC(2026, 7, 1), + overageAllowed: false, + remaining: 2000, + usage: 8000, + }, + })), inserted: [] as Record[], log: { error: vi.fn(), @@ -8,6 +18,14 @@ const state = vi.hoisted(() => ({ warn: vi.fn(), }, operations: [] as string[], + ownedOrganizations: [] as Array<{ + organizationId: string; + organization: { + emailNotifications: { billing?: { usageWarnings?: boolean } }; + id: string; + name: string; + }; + }>, recentRows: [] as Array<{ id: string }>, send: vi.fn(async () => ({ data: { id: "email-1" }, error: null })), userRow: { email: "customer@example.com", name: "Customer" } as { @@ -18,18 +36,22 @@ const state = vi.hoisted(() => ({ vi.mock("@databuddy/db", () => ({ and: (...conditions: unknown[]) => ({ conditions }), - db: { - query: { - member: { findMany: vi.fn(async () => []) }, + db: { + query: { + member: { + findMany: vi.fn(async () => state.ownedOrganizations), + }, organization: { findFirst: vi.fn(async () => null) }, user: { findFirst: vi.fn(async () => state.userRow) }, }, }, eq: (field: unknown, value: unknown) => ({ field, op: "eq", value }), gt: (field: unknown, value: unknown) => ({ field, op: "gt", value }), - normalizeEmailNotificationSettings: () => ({ - billing: { usageWarnings: true }, - }), + normalizeEmailNotificationSettings: (raw?: { + billing?: { usageWarnings?: boolean }; + }) => ({ + billing: { usageWarnings: raw?.billing?.usageWarnings ?? true }, + }), sql: (strings: TemplateStringsArray, ...values: unknown[]) => ({ strings: Array.from(strings), values, @@ -92,6 +114,10 @@ vi.mock("@databuddy/redis", () => ({ invalidateBillingOwnerCaches: vi.fn(async () => ({ attempted: 0, failed: 0 })), })); +vi.mock("@databuddy/rpc", () => ({ + getAutumn: () => ({ check: state.check }), +})); + vi.mock("@databuddy/services/billing-lifecycle", () => ({ recordPlanChange: vi.fn(async () => undefined), })); @@ -126,14 +152,34 @@ vi.mock("../../lib/tracing", () => ({ mergeWideEvent: vi.fn(), })); -import { sendAlertEmail } from "./autumn"; +import { UsageAlertEmail, UsageLimitEmail } from "@databuddy/email"; +import { + handleLimitReached, + handleUsageAlert, + sendAlertEmail, +} from "./autumn"; beforeEach(() => { + process.env.RESEND_API_KEY = "test-resend-key"; state.inserted = []; state.operations = []; + state.ownedOrganizations = []; state.recentRows = []; state.userRow = { email: "customer@example.com", name: "Customer" }; state.send.mockClear(); + state.check.mockClear(); + state.check.mockResolvedValue({ + allowed: true, + balance: { + granted: 10_000, + nextResetAt: Date.UTC(2026, 7, 1), + overageAllowed: false, + remaining: 2000, + usage: 8000, + }, + }); + vi.mocked(UsageAlertEmail).mockClear(); + vi.mocked(UsageLimitEmail).mockClear(); state.send.mockImplementation(async () => { state.operations.push("send"); return { data: { id: "email-1" }, error: null }; @@ -152,6 +198,7 @@ describe("sendAlertEmail", () => { cooldownKey: "events", customerId: "user-1", react: { type: "email" } as never, + recipient: { email: "member@example.com" }, subject: "Limit reached", }); @@ -167,24 +214,227 @@ describe("sendAlertEmail", () => { cooldownKey: "events", customerId: "user-1", react: { type: "email" } as never, + recipient: { email: "member@example.com" }, subject: "Limit reached", }); expect(result).toEqual({ success: true, message: "Email sent" }); expect(state.operations).toEqual(["lock", "select", "send", "insert"]); - expect(state.send).toHaveBeenCalledWith({ - from: "alerts@databuddy.cc", - to: "customer@example.com", - subject: "Limit reached", - html: "", - }); + expect(state.send).toHaveBeenCalledWith({ + from: "alerts@databuddy.cc", + to: "member@example.com", + subject: "Limit reached", + html: "", + text: "", + }); expect(state.inserted).toEqual([ expect.objectContaining({ alertType: "included", - emailSentTo: "customer@example.com", + emailSentTo: "member@example.com", featureId: "events", userId: "user-1", }), ]); }); + + it("does not record a delivery when Resend rejects the email", async () => { + state.send.mockResolvedValueOnce({ + data: null, + error: { message: "provider unavailable" }, + }); + + const result = await sendAlertEmail({ + alertType: "included", + cooldownKey: "events", + customerId: "user-1", + react: { type: "email" } as never, + recipient: { email: "member@example.com" }, + subject: "Limit reached", + }); + + expect(result).toEqual({ + success: false, + message: "Alert email delivery failed", + }); + expect(state.operations).toEqual(["lock", "select"]); + expect(state.inserted).toEqual([]); + }); + + it("reports delivery unavailable when Resend is not configured", async () => { + delete process.env.RESEND_API_KEY; + + const result = await sendAlertEmail({ + alertType: "included", + cooldownKey: "events", + customerId: "user-1", + react: { type: "email" } as never, + recipient: { email: "member@example.com" }, + subject: "Limit reached", + }); + + expect(result).toEqual({ + success: false, + message: "Alert email delivery unavailable", + }); + expect(state.send).not.toHaveBeenCalled(); + }); +}); + +describe("Autumn usage emails", () => { + beforeEach(() => { + state.ownedOrganizations = [ + { + organizationId: "org-1", + organization: { + emailNotifications: { billing: { usageWarnings: true } }, + id: "org-1", + name: "Acme", + }, + }, + ]; + }); + + it("uses live balance data and purpose-based Databunny wording", async () => { + state.userRow = { email: "recipient@example.com", name: "Recipient" }; + state.check.mockResolvedValueOnce({ + allowed: true, + balance: { + granted: 350, + nextResetAt: Date.UTC(2026, 7, 1), + overageAllowed: false, + remaining: 62, + usage: 288, + }, + }); + + await handleUsageAlert({ + customer_id: "user-1", + entity_id: "org-1", + feature_id: "agent_credits", + usage_alert: { + name: "AI notice", + threshold: 80, + threshold_type: "usage_percentage", + }, + }); + + expect(UsageAlertEmail).toHaveBeenCalledWith( + expect.objectContaining({ + featureName: "Databunny usage", + limitAmount: 350, + organizationName: "Acme", + remainingAmount: 62, + usageAmount: 288, + usageUnit: "allowance units", + }) + ); + expect(UsageAlertEmail).not.toHaveBeenCalledWith( + expect.objectContaining({ userName: expect.anything() }) + ); + expect(state.send).toHaveBeenCalledWith( + expect.objectContaining({ + subject: "Databunny usage is at 82%", + to: "recipient@example.com", + }) + ); + }); + + it("handles an actual hard limit and tells the template whether use is paused", async () => { + state.check.mockResolvedValueOnce({ + allowed: false, + balance: { + granted: 350, + nextResetAt: Date.UTC(2026, 7, 1), + overageAllowed: false, + remaining: 0, + usage: 350, + }, + }); + + await handleLimitReached({ + customer_id: "user-1", + entity_id: "org-1", + feature_id: "agent_credits", + limit_type: "spend_limit", + }); + + expect(UsageLimitEmail).toHaveBeenCalledWith( + expect.objectContaining({ + featureName: "Databunny usage", + isAvailable: false, + limitAmount: 350, + limitType: "spend_limit", + usageAmount: 350, + }) + ); + expect(state.send).toHaveBeenCalledWith( + expect.objectContaining({ + subject: "[Action required] Databunny usage is paused", + }) + ); + }); + + it("honors the resolved organization's billing email preference", async () => { + state.ownedOrganizations[0]!.organization.emailNotifications = { + billing: { usageWarnings: false }, + }; + + const result = await handleLimitReached({ + customer_id: "user-1", + entity_id: "org-1", + feature_id: "events", + limit_type: "included", + }); + + expect(result).toEqual({ + success: true, + message: "Billing usage emails disabled", + }); + expect(state.send).not.toHaveBeenCalled(); + }); + + it("defers usage alerts when multiple organizations are ambiguous", async () => { + state.ownedOrganizations.push({ + organizationId: "org-2", + organization: { + emailNotifications: { billing: { usageWarnings: true } }, + id: "org-2", + name: "Second organization", + }, + }); + + const result = await handleUsageAlert({ + customer_id: "user-1", + feature_id: "events", + usage_alert: { + threshold: 80, + threshold_type: "usage_percentage", + }, + }); + + expect(result).toEqual({ + success: false, + message: "Billing usage email deferred: organization could not be resolved", + }); + expect(state.check).not.toHaveBeenCalled(); + expect(UsageAlertEmail).not.toHaveBeenCalled(); + expect(state.send).not.toHaveBeenCalled(); + }); + + it("defers limit alerts when the entity does not resolve", async () => { + const result = await handleLimitReached({ + customer_id: "user-1", + entity_id: "org-missing", + feature_id: "events", + limit_type: "included", + }); + + expect(result).toEqual({ + success: false, + message: "Billing usage email deferred: organization could not be resolved", + }); + expect(state.check).not.toHaveBeenCalled(); + expect(UsageLimitEmail).not.toHaveBeenCalled(); + expect(state.send).not.toHaveBeenCalled(); + }); }); diff --git a/apps/api/src/routes/webhooks/autumn.ts b/apps/api/src/routes/webhooks/autumn.ts index c41653c300..7c243e96fc 100644 --- a/apps/api/src/routes/webhooks/autumn.ts +++ b/apps/api/src/routes/webhooks/autumn.ts @@ -17,6 +17,7 @@ import { invalidateAgentContextSnapshotsForOwner, invalidateBillingOwnerCaches, } from "@databuddy/redis"; +import { getAutumn } from "@databuddy/rpc"; import { recordPlanChange } from "@databuddy/services/billing-lifecycle"; import { Elysia } from "elysia"; import { useLogger } from "evlog/elysia"; @@ -31,18 +32,19 @@ const COOLDOWN_MS = 7 * 24 * 60 * 60 * 1000; const SVIX_SECRET = process.env.AUTUMN_WEBHOOK_SECRET; const SLACK_URL = process.env.SLACK_WEBHOOK_URL ?? ""; -const resend = new Resend(process.env.RESEND_API_KEY); const svix = SVIX_SECRET ? new Webhook(SVIX_SECRET) : null; const slack = SLACK_URL ? new SlackProvider({ webhookUrl: SLACK_URL }) : null; const limitReachedSchema = z.object({ customer_id: z.string(), + entity_id: z.string().optional(), feature_id: z.string(), limit_type: z.enum(["included", "max_purchase", "spend_limit"]), }); const usageAlertSchema = z.object({ customer_id: z.string(), + entity_id: z.string().optional(), feature_id: z.string(), usage_alert: z.object({ name: z.string().optional(), @@ -92,23 +94,39 @@ interface WebhookResult { success: boolean; } -async function getOrganizationEmailSettings(customerId: string) { - const row = await db.query.organization.findFirst({ - where: { id: customerId }, - columns: { emailNotifications: true }, - }); - return normalizeEmailNotificationSettings(row?.emailNotifications); +interface BillingRecipient { + email: string | null; +} + +interface BillingOrganizationContext { + emailNotifications: Parameters[0]; + id: string; + name: string; +} + +interface UsageSnapshot { + granted: number; + isAvailable: boolean; + nextResetAt: number | null; + overageAllowed: boolean; + remaining: number; + usage: number; +} + +interface BillingFeatureCopy { + description: string; + name: string; + pausedActivity: string; + unit: string; } -const getUserData = cacheable( - async ( - customerId: string - ): Promise<{ email: string | null; name: string | null }> => { +const getBillingRecipient = cacheable( + async (customerId: string): Promise => { const row = await db.query.user.findFirst({ where: { id: customerId }, - columns: { email: true, name: true }, + columns: { email: true }, }); - return { email: row?.email ?? null, name: row?.name ?? null }; + return { email: row?.email ?? null }; }, { expireInSec: 300, @@ -118,8 +136,105 @@ const getUserData = cacheable( } ); -function formatFeatureId(id: string): string { - return id.replace(/_/g, " ").replace(/\b\w/g, (c) => c.toUpperCase()); +export async function resolveBillingOrganization( + customerId: string, + entityId?: string +): Promise { + const memberships = await db.query.member.findMany({ + where: { userId: customerId, role: "owner" }, + columns: { organizationId: true }, + with: { + organization: { + columns: { id: true, name: true, emailNotifications: true }, + }, + }, + }); + + const candidates = entityId + ? memberships.filter((row) => row.organizationId === entityId) + : memberships; + if (candidates.length !== 1) { + return null; + } + + const organization = candidates[0]?.organization; + return organization + ? { + emailNotifications: organization.emailNotifications, + id: organization.id, + name: organization.name, + } + : null; +} + +function getFeatureCopy(featureId: string): BillingFeatureCopy { + if (featureId === "agent_credits") { + return { + description: + "This allowance powers Databunny questions and AI analysis. More complex work uses more of it.", + name: "Databunny usage", + pausedActivity: "Databunny questions and AI analysis", + unit: "allowance units", + }; + } + + if (featureId === "events") { + return { + description: + "Events include page views, custom events, errors, and Web Vitals collected by Databuddy.", + name: "Event tracking", + pausedActivity: "new event collection", + unit: "events", + }; + } + + const label = featureId.replaceAll(/[_-]+/g, " ").trim() || "Feature"; + return { + description: `This allowance controls how much ${label} your plan can use.`, + name: `${label[0]?.toUpperCase() ?? ""}${label.slice(1)} usage`, + pausedActivity: label, + unit: "units", + }; +} + +async function getUsageSnapshot( + customerId: string, + featureId: string +): Promise { + try { + const response = await getAutumn().check({ customerId, featureId }); + const balance = response.balance; + if (!balance) { + return null; + } + return { + granted: balance.granted, + isAvailable: response.allowed, + nextResetAt: balance.nextResetAt, + overageAllowed: balance.overageAllowed, + remaining: balance.remaining, + usage: balance.usage, + }; + } catch (error) { + useLogger().error( + error instanceof Error ? error : new Error(String(error)), + { autumn: { step: "load_usage", customerId, featureId } } + ); + return null; + } +} + +function formatUsageNumber(value: number): string { + return new Intl.NumberFormat("en-US", { maximumFractionDigits: 1 }).format( + value + ); +} + +function usagePercentage(snapshot: UsageSnapshot): number | null { + if (snapshot.granted <= 0) { + return null; + } + return Math.round((snapshot.usage / snapshot.granted) * 100); } export async function sendAlertEmail(opts: { @@ -128,19 +243,29 @@ export async function sendAlertEmail(opts: { alertType: string; subject: string; react: React.ReactElement; + recipient: BillingRecipient; }): Promise { const log = useLogger(); - const { customerId, cooldownKey, alertType, subject, react } = opts; + const { customerId, cooldownKey, alertType, subject, react, recipient } = + opts; - const { email } = await getUserData(customerId); + const { email } = recipient; if (!email) { log.warn("No email for customer", { autumn: { customerId, cooldownKey }, }); return { success: true, message: "No notification recipient found" }; } + const resendApiKey = process.env.RESEND_API_KEY; + if (!resendApiKey) { + log.error(new Error("RESEND_API_KEY is not configured"), { + autumn: { customerId, cooldownKey, step: "email_configuration" }, + }); + return { success: false, message: "Alert email delivery unavailable" }; + } + const resend = new Resend(resendApiKey); - return withTransaction(async (tx) => { + return await withTransaction(async (tx) => { await tx.execute( sql`SELECT pg_advisory_xact_lock(hashtextextended(${`usage-alert:${customerId}:${cooldownKey}`}, 0))` ); @@ -165,12 +290,16 @@ export async function sendAlertEmail(opts: { return { success: true, message: "Already sent recently" }; } - const html = await render(react); + const [html, text] = await Promise.all([ + render(react), + render(react, { plainText: true }), + ]); const result = await resend.emails.send({ from: config.email.alertsFrom, to: email, subject, html, + text, }); if (result.error) { @@ -221,42 +350,103 @@ async function invalidatePlanCaches(customerId: string | null): Promise { } } -function handleLimitReached( +export async function handleLimitReached( data: LimitReachedData -): Promise | WebhookResult { - const { customer_id, feature_id, limit_type } = data; +): Promise { + const { customer_id, entity_id, feature_id, limit_type } = data; + const organization = await resolveBillingOrganization(customer_id, entity_id); + if (!organization) { + useLogger().warn("Could not resolve billing organization", { + autumn: { customerId: customer_id, entityId: entity_id }, + }); + return { + success: false, + message: + "Billing usage email deferred: organization could not be resolved", + }; + } - if (limit_type !== "included") { - return { success: true, message: `Skipped ${limit_type} limit` }; + const [recipient, snapshot] = await Promise.all([ + getBillingRecipient(customer_id), + getUsageSnapshot(customer_id, feature_id), + ]); + + if ( + !normalizeEmailNotificationSettings(organization.emailNotifications).billing + .usageWarnings + ) { + return { success: true, message: "Billing usage emails disabled" }; + } + if (!snapshot) { + return { success: false, message: "Current usage is unavailable" }; } - const featureName = formatFeatureId(feature_id); + const feature = getFeatureCopy(feature_id); + const isHardStop = !snapshot.isAvailable; + const subject = isHardStop + ? `[Action required] ${feature.name} is paused` + : `${feature.name}: included allowance used`; mergeWideEvent({ customer_id, feature_id, limit_type }); return sendAlertEmail({ customerId: customer_id, - cooldownKey: feature_id, + cooldownKey: `${feature_id}:limit:${limit_type}`, alertType: limit_type, - subject: `[Action required] ${featureName} limit reached — upgrade to continue tracking`, + subject, react: UsageLimitEmail({ - featureName, - thresholdType: "limit_reached", + featureDescription: feature.description, + featureName: feature.name, + isAvailable: snapshot.isAvailable, + limitAmount: snapshot.granted, + limitType: limit_type, + nextResetAt: snapshot.nextResetAt, + organizationName: organization?.name, + overageAllowed: snapshot.overageAllowed, + pausedActivity: feature.pausedActivity, + remainingAmount: snapshot.remaining, + usageAmount: snapshot.usage, + usageUnit: feature.unit, }), + recipient, }); } -async function handleUsageAlert(data: UsageAlertData): Promise { - const { customer_id, feature_id, usage_alert } = data; - const settings = await getOrganizationEmailSettings(customer_id); - if (!settings.billing.usageWarnings) { - return { success: true, message: "Usage warning emails disabled" }; +export async function handleUsageAlert( + data: UsageAlertData +): Promise { + const { customer_id, entity_id, feature_id, usage_alert } = data; + const organization = await resolveBillingOrganization(customer_id, entity_id); + if (!organization) { + useLogger().warn("Could not resolve billing organization", { + autumn: { customerId: customer_id, entityId: entity_id }, + }); + return { + success: false, + message: + "Billing usage email deferred: organization could not be resolved", + }; + } + + const [recipient, snapshot] = await Promise.all([ + getBillingRecipient(customer_id), + getUsageSnapshot(customer_id, feature_id), + ]); + if ( + !normalizeEmailNotificationSettings(organization.emailNotifications).billing + .usageWarnings + ) { + return { success: true, message: "Billing usage emails disabled" }; } - const featureName = formatFeatureId(feature_id); - const isPercentage = - usage_alert.threshold_type === "usage_percentage_threshold"; - const label = isPercentage - ? `${usage_alert.threshold}%` - : String(usage_alert.threshold); + if (!snapshot) { + return { success: false, message: "Current usage is unavailable" }; + } + + const feature = getFeatureCopy(feature_id); + const percentage = usagePercentage(snapshot); + const subject = + percentage === null + ? `${feature.name}: ${formatUsageNumber(snapshot.usage)} ${feature.unit} used` + : `${feature.name} is at ${percentage}%`; mergeWideEvent({ customer_id, @@ -267,15 +457,22 @@ async function handleUsageAlert(data: UsageAlertData): Promise { return sendAlertEmail({ customerId: customer_id, - cooldownKey: `${feature_id}_alert_${usage_alert.threshold}`, + cooldownKey: `${feature_id}:alert:${usage_alert.threshold_type}:${usage_alert.threshold}`, alertType: `usage_alert_${usage_alert.threshold_type}`, - subject: `[Action required] You've used ${label} of your ${featureName.toLowerCase()}`, + subject, react: UsageAlertEmail({ - featureName, - threshold: usage_alert.threshold, - thresholdType: isPercentage ? "usage_percentage_threshold" : "usage", - alertName: usage_alert.name ?? undefined, + featureDescription: feature.description, + featureName: feature.name, + limitAmount: snapshot.granted, + nextResetAt: snapshot.nextResetAt, + organizationName: organization?.name, + overageAllowed: snapshot.overageAllowed, + pausedActivity: feature.pausedActivity, + remainingAmount: snapshot.remaining, + usageAmount: snapshot.usage, + usageUnit: feature.unit, }), + recipient, }); } diff --git a/apps/api/src/rpc/handlers.ts b/apps/api/src/rpc/handlers.ts index b5d427831d..fa3ec8aef4 100644 --- a/apps/api/src/rpc/handlers.ts +++ b/apps/api/src/rpc/handlers.ts @@ -8,6 +8,7 @@ import { ORPCError, onError } from "@orpc/server"; import { RPCHandler } from "@orpc/server/fetch"; import { useLogger } from "evlog/elysia"; import { getResolvedAuth } from "@/lib/auth-wide-event"; +import { getRequestId } from "@/http/request-id"; import { logOrpcHandlerError } from "./interceptors"; export type OrpcContext = Awaited>; @@ -51,6 +52,7 @@ async function handleOrpcRequest( createContext: (request: Request) => Promise, handle: OrpcRouteHandler ) { + const requestId = getRequestId(request); try { const context = await createContext(request); const result = await handle(request, context); @@ -61,8 +63,9 @@ async function handleOrpcRequest( success: false, error: "Not found", code: "NOT_FOUND", + requestId, }, - { status: 404 } + { status: 404, headers: { "X-Request-ID": requestId } } ) ); } catch (error) { @@ -78,8 +81,9 @@ async function handleOrpcRequest( success: false, error: "An internal server error occurred", code: "INTERNAL_SERVER_ERROR", + requestId, }, - { status: 500 } + { status: 500, headers: { "X-Request-ID": requestId } } ); } } diff --git a/apps/api/src/rpc/openapi-config.ts b/apps/api/src/rpc/openapi-config.ts index e13d47e8e3..92aab73941 100644 --- a/apps/api/src/rpc/openapi-config.ts +++ b/apps/api/src/rpc/openapi-config.ts @@ -39,7 +39,7 @@ export const OPENAPI_TAGS = [ { name: "Alarms", description: - "Alert rules and notifications for metrics and conditions across your workspace.", + "Alert rules and notifications for metrics and conditions across your organization.", }, { name: "Annotations", @@ -58,7 +58,8 @@ export const OPENAPI_TAGS = [ }, { name: "Feedback", - description: "Submit and manage product feedback tied to your workspace.", + description: + "Submit and manage product feedback tied to your organization.", }, { name: "Flags", @@ -83,7 +84,7 @@ export const OPENAPI_TAGS = [ { name: "Organizations", description: - "Workspace and organization management: avatar, invitations, billing context, and usage.", + "Organization management: avatar, invitations, billing context, and usage.", }, { name: "Preferences", @@ -108,6 +109,6 @@ export const OPENAPI_TAGS = [ { name: "Websites", description: - "Website management: create, list, update, delete websites; transfer between workspaces; configure settings, tracking, and data export.", + "Website management: create, list, update, delete websites; transfer between organizations; configure settings, tracking, and data export.", }, ] as const; diff --git a/apps/api/src/types/tables.ts b/apps/api/src/types/tables.ts deleted file mode 100644 index 4899ef6721..0000000000 --- a/apps/api/src/types/tables.ts +++ /dev/null @@ -1,38 +0,0 @@ -import type { - AnalyticsEvent, - BlockedTraffic, - CustomEvent, - CustomOutgoingLink, - ErrorSpanRow, - RevenueTransaction, - UptimeMonitor, - WebVitalsHourlyAggregate, - WebVitalsSpan, -} from "@databuddy/db/clickhouse/schema"; - -export const Analytics = { - events: "analytics.events", - error_spans: "analytics.error_spans", - web_vitals_spans: "analytics.web_vitals_spans", - web_vitals_hourly: "analytics.web_vitals_hourly", - custom_events: "analytics.custom_events", - blocked_traffic: "analytics.blocked_traffic", - outgoing_links: "analytics.outgoing_links", - link_visits: "analytics.link_visits", - uptime_monitor: "uptime.uptime_monitor", - revenue: "analytics.revenue", -} as const; - -export type AnalyticsTable = (typeof Analytics)[keyof typeof Analytics]; - -export interface TableFieldsMap { - "analytics.blocked_traffic": keyof BlockedTraffic; - "analytics.custom_events": keyof CustomEvent; - "analytics.error_spans": keyof ErrorSpanRow; - "analytics.events": keyof AnalyticsEvent; - "analytics.outgoing_links": keyof CustomOutgoingLink; - "analytics.revenue": keyof RevenueTransaction; - "analytics.web_vitals_hourly": keyof WebVitalsHourlyAggregate; - "analytics.web_vitals_spans": keyof WebVitalsSpan; - "uptime.uptime_monitor": keyof UptimeMonitor; -} diff --git a/apps/basket/src/hooks/auth.ts b/apps/basket/src/hooks/auth.ts index f3b2c6705e..885ff0d8f2 100644 --- a/apps/basket/src/hooks/auth.ts +++ b/apps/basket/src/hooks/auth.ts @@ -9,6 +9,7 @@ import { db } from "@databuddy/db"; import type { Website } from "@databuddy/db/schema"; import { cacheNamespaces } from "@databuddy/redis/cache-invalidation"; import { cacheable } from "@databuddy/redis/cacheable"; +import { basketErrors } from "@lib/structured-errors"; import { captureError, record } from "@lib/tracing"; import { isValidOriginFromSettings } from "@utils/origin-ip-validation"; import { createError, EvlogError } from "evlog"; @@ -45,10 +46,14 @@ function _resolveOwnerId( return orgMember.userId; } } catch (error) { + if (error instanceof EvlogError) { + throw error; + } captureError(error, { message: "Failed to fetch workspace owner", organizationId, }); + throw basketErrors.websiteLookupUnavailable(); } return null; @@ -198,11 +203,14 @@ const getWebsiteByIdWithOwnerCached = cacheable( const ownerId = await _resolveOwnerId(website.organizationId); return { ...website, ownerId }; } catch (error) { + if (error instanceof EvlogError) { + throw error; + } captureError(error, { message: "Failed to get website by ID from cache", websiteId: id, }); - return null; + throw basketErrors.websiteLookupUnavailable(); } }, { @@ -233,11 +241,14 @@ export function getWebsiteByIdV2(id: string): Promise { try { return await getWebsiteByIdWithOwnerCached(id); } catch (error) { + if (error instanceof EvlogError) { + throw error; + } captureError(error, { message: "Failed to get website by ID V2", websiteId: id, }); - return null; + throw basketErrors.websiteLookupUnavailable(); } }); } diff --git a/apps/basket/src/index.ts b/apps/basket/src/index.ts index f87aaf0565..1851329005 100644 --- a/apps/basket/src/index.ts +++ b/apps/basket/src/index.ts @@ -98,16 +98,20 @@ const app = new Elysia() set.headers["Access-Control-Allow-Credentials"] = "true"; } }) - .onError(function handleError({ error, code }) { + .onError(function handleError({ error, code, request, set }) { if (code === "NOT_FOUND") { return new Response(null, { status: 404 }); } - captureError(error); + const requestId = + request.headers.get("x-request-id") ?? crypto.randomUUID(); + captureError(error, { requestId }); const { status, payload } = buildBasketErrorPayload(error, { elysiaCode: code ?? "INTERNAL_SERVER_ERROR", + extra: { requestId }, }); + set.headers["x-request-id"] = requestId; return new Response(JSON.stringify(payload), { status, diff --git a/apps/basket/src/lib/blocked-traffic-alerts.test.ts b/apps/basket/src/lib/blocked-traffic-alerts.test.ts index 302c3671b5..62d6173206 100644 --- a/apps/basket/src/lib/blocked-traffic-alerts.test.ts +++ b/apps/basket/src/lib/blocked-traffic-alerts.test.ts @@ -2,6 +2,7 @@ import { describe, expect, test } from "vitest"; import { decideBlockedTrafficAlert, matchesTrackingAlertIgnoredOrigin, + requireBlockedTrafficEmailApiKey, shouldEvaluateBlockedTrafficAlert, shouldIgnoreBlockedTrafficAlertEvent, } from "./blocked-traffic-alerts"; @@ -148,4 +149,11 @@ describe("blocked traffic alert rules", () => { expect(shouldEvaluateBlockedTrafficAlert(50)).toBe(true); expect(shouldEvaluateBlockedTrafficAlert(75)).toBe(true); }); + + test("fails delivery when the email provider is not configured", () => { + expect(() => requireBlockedTrafficEmailApiKey(undefined)).toThrow( + "Blocked traffic email delivery is not configured" + ); + expect(requireBlockedTrafficEmailApiKey("test-key")).toBe("test-key"); + }); }); diff --git a/apps/basket/src/lib/blocked-traffic-alerts.ts b/apps/basket/src/lib/blocked-traffic-alerts.ts index 90547495d2..ce50326f58 100644 --- a/apps/basket/src/lib/blocked-traffic-alerts.ts +++ b/apps/basket/src/lib/blocked-traffic-alerts.ts @@ -1,4 +1,4 @@ -import type { BlockedTraffic } from "@databuddy/db/clickhouse/schema"; +import type { BlockedTrafficInsert } from "@databuddy/db/clickhouse/tables"; import { chQuery } from "@databuddy/db/clickhouse"; import { db, @@ -45,16 +45,16 @@ export interface BlockedTrafficAlertDecision { severity: "critical" | "warning"; } -function getAlertOrigin(event: BlockedTraffic): string { +function getAlertOrigin(event: BlockedTrafficInsert): string { return event.origin?.trim() || ""; } -function getAlertOriginKey(event: BlockedTraffic): string { +function getAlertOriginKey(event: BlockedTrafficInsert): string { return encodeURIComponent(getAlertOrigin(event) || "missing-origin"); } function getBlockedTrafficSource( - event: Pick + event: Pick ): string | null { return event.origin || event.referrer || null; } @@ -68,7 +68,7 @@ export function matchesTrackingAlertIgnoredOrigin( export function shouldIgnoreBlockedTrafficAlertEvent( event: Pick< - BlockedTraffic, + BlockedTrafficInsert, "block_reason" | "client_id" | "origin" | "referrer" > ): boolean { @@ -81,7 +81,7 @@ export function shouldIgnoreBlockedTrafficAlertEvent( return isIgnoredTrackingBlockOrigin(getBlockedTrafficSource(event)); } -function buildRecommendedFix(event: BlockedTraffic): string { +function buildRecommendedFix(event: BlockedTrafficInsert): string { const host = getTrackingBlockOriginHost(event.origin ?? null); if (event.block_reason === "origin_not_authorized") { return host @@ -101,7 +101,9 @@ function buildDashboardUrl(clientId: string, reason: string): string { return `${config.urls.dashboard}/websites/${clientId}/settings/${section}`; } -async function incrementWindowCounter(event: BlockedTraffic): Promise { +async function incrementWindowCounter( + event: BlockedTrafficInsert +): Promise { const key = `blocked-traffic-alert:count:${event.client_id}:${event.block_reason}:${getAlertOriginKey(event)}`; const count = await redis.incr(key); if (count === 1) { @@ -124,7 +126,9 @@ async function getTrackingHealth( return rows[0] ?? { baselineEvents: 0, recentEvents: 0 }; } -async function getPreviousBlockedCount(event: BlockedTraffic): Promise { +async function getPreviousBlockedCount( + event: BlockedTrafficInsert +): Promise { const rows = await chQuery( `SELECT count() AS previousBlocked FROM analytics.blocked_traffic @@ -176,15 +180,24 @@ export function shouldEvaluateBlockedTrafficAlert( ); } +export function requireBlockedTrafficEmailApiKey( + apiKey: string | undefined +): string { + if (!apiKey) { + throw new Error("Blocked traffic email delivery is not configured"); + } + return apiKey; +} + function cooldownKey( - event: BlockedTraffic, + event: BlockedTrafficInsert, kind: BlockedTrafficAlertDecision["kind"] ): string { return `blocked-traffic-alert:sent:${event.client_id}:${event.block_reason}:${getAlertOriginKey(event)}:${kind}`; } async function reserveCooldown( - event: BlockedTraffic, + event: BlockedTrafficInsert, kind: BlockedTrafficAlertDecision["kind"], settings: EmailNotificationSettings ): Promise { @@ -194,17 +207,15 @@ async function reserveCooldown( return reserved === "OK" ? key : null; } -async function getOwnerEmail( - ownerId: string -): Promise<{ email: string; name: string | null } | null> { +async function getOwnerEmail(ownerId: string): Promise { const row = await db.query.user.findFirst({ where: { id: ownerId }, - columns: { email: true, name: true }, + columns: { email: true }, }); if (!row?.email) { return null; } - return { email: row.email, name: row.name ?? null }; + return row.email; } async function getOrganizationEmailSettings( @@ -223,7 +234,7 @@ async function getOrganizationEmailSettings( function isAlertMutedBySettings(input: { decision: BlockedTrafficAlertDecision; - event: BlockedTraffic; + event: BlockedTrafficInsert; settings: EmailNotificationSettings; }): boolean { const tracking = input.settings.trackingHealth; @@ -252,16 +263,13 @@ function isAlertMutedBySettings(input: { async function sendAlertEmail(input: { context: BlockedTrafficAlertContext; decision: BlockedTrafficAlertDecision; - event: BlockedTraffic; + event: BlockedTrafficInsert; trackingHealth: TrackingHealthCounts; - ownerEmail: string; + recipientEmail: string; previousBlocked: number; windowBlockedCount: number; }): Promise { - const apiKey = process.env.RESEND_API_KEY; - if (!apiKey) { - return; - } + const apiKey = requireBlockedTrafficEmailApiKey(process.env.RESEND_API_KEY); const siteLabel = input.context.websiteName || @@ -273,25 +281,27 @@ async function sendAlertEmail(input: { ? `[Action required] Tracking may be blocked for ${siteLabel}` : `[Databuddy] Blocked tracking increased for ${siteLabel}`; - const html = await render( - BlockedTrafficAlertEmail({ - baselineEvents: input.trackingHealth.baselineEvents, - baselineHours: BASELINE_SUCCESS_HOURS, - blockReason: input.event.block_reason, - blockedCount: input.windowBlockedCount, - dashboardUrl: buildDashboardUrl( - input.event.client_id || "", - input.event.block_reason - ), - fix: buildRecommendedFix(input.event), - origin: input.event.origin ?? null, - previousBlockedCount: input.previousBlocked, - recentEvents: input.trackingHealth.recentEvents, - severity: input.decision.severity, - siteLabel, - windowMinutes: ALERT_WINDOW_MINUTES, - }) - ); + const email = BlockedTrafficAlertEmail({ + baselineEvents: input.trackingHealth.baselineEvents, + baselineHours: BASELINE_SUCCESS_HOURS, + blockReason: input.event.block_reason, + blockedCount: input.windowBlockedCount, + dashboardUrl: buildDashboardUrl( + input.event.client_id || "", + input.event.block_reason + ), + fix: buildRecommendedFix(input.event), + origin: input.event.origin ?? null, + previousBlockedCount: input.previousBlocked, + recentEvents: input.trackingHealth.recentEvents, + severity: input.decision.severity, + siteLabel, + windowMinutes: ALERT_WINDOW_MINUTES, + }); + const [html, text] = await Promise.all([ + render(email), + render(email, { plainText: true }), + ]); const response = await fetch("https://api.resend.com/emails", { method: "POST", @@ -301,9 +311,10 @@ async function sendAlertEmail(input: { }, body: JSON.stringify({ from: config.email.alertsFrom, - to: input.ownerEmail, + to: input.recipientEmail, subject, html, + text, }), }); @@ -313,7 +324,7 @@ async function sendAlertEmail(input: { } async function maybeSendBlockedTrafficAlertAsync( - event: BlockedTraffic, + event: BlockedTrafficInsert, context: BlockedTrafficAlertContext = {} ): Promise { if (shouldIgnoreBlockedTrafficAlertEvent(event)) { @@ -350,8 +361,8 @@ async function maybeSendBlockedTrafficAlertAsync( return; } - const owner = await getOwnerEmail(context.ownerId); - if (!owner) { + const recipientEmail = await getOwnerEmail(context.ownerId); + if (!recipientEmail) { return; } @@ -366,7 +377,7 @@ async function maybeSendBlockedTrafficAlertAsync( decision, event, trackingHealth, - ownerEmail: owner.email, + recipientEmail, previousBlocked, windowBlockedCount, }); @@ -377,7 +388,7 @@ async function maybeSendBlockedTrafficAlertAsync( } export function queueBlockedTrafficAlert( - event: BlockedTraffic, + event: BlockedTrafficInsert, context?: BlockedTrafficAlertContext ): void { maybeSendBlockedTrafficAlertAsync(event, context).catch((error) => { diff --git a/apps/basket/src/lib/blocked-traffic.ts b/apps/basket/src/lib/blocked-traffic.ts index 7466c7f2e1..62ca47d3dd 100644 --- a/apps/basket/src/lib/blocked-traffic.ts +++ b/apps/basket/src/lib/blocked-traffic.ts @@ -1,4 +1,4 @@ -import type { BlockedTraffic } from "@databuddy/db/clickhouse/schema"; +import type { BlockedTrafficInsert } from "@databuddy/db/clickhouse/tables"; import { queueBlockedTrafficAlert, type BlockedTrafficAlertContext, @@ -37,10 +37,10 @@ async function _logBlockedTrafficAsync( parseUserAgent(userAgent), ]); const now = Date.now(); - const { anonymizedIP, country, region, city } = geo; + const { anonymizedIP, country, region } = geo; const { browserName, browserVersion, osName, osVersion, deviceType } = ua; - const blockedEvent: BlockedTraffic = { + const blockedEvent: BlockedTrafficInsert = { id: randomUUIDv7(), client_id: clientId || "", timestamp: now, @@ -77,7 +77,6 @@ async function _logBlockedTrafficAsync( country: country || "", region: region || "", - city: city || "", browser_name: browserName || "", browser_version: browserVersion || "", os_name: osName || "", diff --git a/apps/basket/src/lib/event-service.test.ts b/apps/basket/src/lib/event-service.test.ts index 72cec321e2..353fe536cd 100644 --- a/apps/basket/src/lib/event-service.test.ts +++ b/apps/basket/src/lib/event-service.test.ts @@ -74,8 +74,6 @@ describe("buildTrackEvent — field mapping", () => { // Identity expect(result.id).toBeTruthy(); // randomUUIDv7 expect(result.client_id).toBe("ws_test"); - expect(result.event_id).toBe("evt_123"); - expect(result.event_type).toBe("track"); // Names & content expect(result.event_name).toBe("pageview"); @@ -91,7 +89,6 @@ describe("buildTrackEvent — field mapping", () => { // Timestamps — uses trackData values when numeric expect(result.timestamp).toBe(1_700_000_001_000); expect(result.time).toBe(1_700_000_001_000); - expect(result.session_start_time).toBe(1_700_000_000_500); expect(result.created_at).toBe(NOW); // Geo @@ -111,13 +108,9 @@ describe("buildTrackEvent — field mapping", () => { expect(result.device_model).toBe("XPS"); // Client context — passthrough - expect(result.screen_resolution).toBe("1920x1080"); expect(result.viewport_size).toBe("1024x768"); expect(result.language).toBe("en-US"); expect(result.timezone).toBe("America/New_York"); - expect(result.connection_type).toBe("wifi"); - expect(result.rtt).toBe(50); - expect(result.downlink).toBe(10.5); // Engagement expect(result.time_on_page).toBe(30_000); @@ -134,14 +127,9 @@ describe("buildTrackEvent — field mapping", () => { expect(result.gclid).toBe("gclid_abc"); // Performance — validated through validatePerformanceMetric - expect(result.load_time).toBe(1500); expect(result.dom_ready_time).toBe(800); - expect(result.dom_interactive).toBe(600); expect(result.ttfb).toBe(200); - expect(result.connection_time).toBe(50); expect(result.render_time).toBe(100); - expect(result.redirect_time).toBe(10); - expect(result.domain_lookup_time).toBe(30); // Properties expect(result.properties).toBe('{"plan":"pro","color":"blue"}'); @@ -153,7 +141,6 @@ describe("buildTrackEvent — field mapping", () => { expect(result.event_name).toBe("click"); expect(result.timestamp).toBe(NOW); // falls back to ctx.now expect(result.time).toBe(NOW); - expect(result.session_start_time).toBe(NOW); expect(result.page_count).toBe(1); // default expect(result.properties).toBe("{}"); // empty expect(result.referrer).toBe(""); @@ -186,15 +173,10 @@ describe("buildTrackEvent — field mapping", () => { fullCtx ); expect(result.timestamp).toBe(NOW); - expect(result.session_start_time).toBe(NOW); }); test("performance metrics validated (negative → undefined)", () => { - const result = buildTrackEvent( - { name: "x", load_time: -1, ttfb: 999_999 }, - fullCtx - ); - expect(result.load_time).toBeUndefined(); + const result = buildTrackEvent({ name: "x", ttfb: 999_999 }, fullCtx); expect(result.ttfb).toBeUndefined(); // >300000 }); @@ -282,20 +264,17 @@ describe("buildTrackEvent — sanitization boundary", () => { expect(() => JSON.parse(result.properties as string)).not.toThrow(); }); - test("passthrough fields (screen_resolution, language, etc.) are NOT sanitized", () => { + test("passthrough fields (language, timezone, etc.) are NOT sanitized", () => { const result = buildTrackEvent( { name: "x", - screen_resolution: "`} /> @@ -91,8 +91,7 @@ Add the `` component once in your root layout: {children} @@ -130,7 +129,6 @@ Add the script to your HTML before the closing `` tag: