perf(traces): serve the explore trace list from derived rollup tables#136
perf(traces): serve the explore trace list from derived rollup tables#136arseniycodes wants to merge 2 commits into
Conversation
The explore "traces" grouped list groups raw otel_traces by TraceId over the
whole selected window, then sorts and limits. otel_traces is sorted by
(ServiceName, SpanName, Timestamp) and the project filter lives in the
ResourceAttributes map, so a project's list can't use the primary index — it
scans the entire multi-tenant window. On a busy deployment a 24h range reads
hundreds of millions of rows and exceeds the ClickHouse request timeout, so the
list 500s while the (rollup-backed) chart above it still renders.
Add two derived tables (migration 005) and read the unfiltered list in two
steps, the way trace backends do:
1. otel_traces_recent — a plain, time-ordered span index
(project_id, ts, trace_id). A bounded reverse scan of the newest spans,
grouped by trace_id, yields the N most recently started trace_ids in
milliseconds. Covers all traces (no root-span dependency).
2. otel_traces_summary — one AggregatingMergeTree row per (project_id,
trace_id) with start, span/error counts, root span/service/status,
distinct-service count and the [start,end] nanos for total duration.
Looked up for just those N trace_ids (IN, indexed) to fill in the stats.
queryTracesAggregated uses this path only for the unfiltered list (the default
view that times out); any span-selecting filter or a duration floor falls back
to the raw scan. Availability is probed once per client and memoized, so
deployments without the tables (e.g. local dev, or before the migration lands)
fall back automatically.
Measured on a 7.8M-trace/24h project: ~1s vs 15-60s, and the two-step result
matches the raw ground truth exactly (top-100 identical, all fields equal).
The materialized views only capture spans ingested after creation; historical
traces need the one-shot backfill script
apps/worker/scripts/backfill-otel-traces-summary.ts, run over
[retention_start, mv_cutover) so it never overlaps the live MV window.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 4 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 80b7e5d. Configure here.
| WHERE project_id = {projectId:String} | ||
| AND trace_id IN (recent_ids) | ||
| AND start >= ${sinceExpr} | ||
| AND start <= ${untilExpr} |
There was a problem hiding this comment.
Summary start filter drops traces
High Severity
The fast-path summary step filters on global trace start in the selected range, but the raw list includes any trace with spans in the window and uses window-local min(Timestamp) for start_time. Traces that started before since yet still have spans in range are dropped from the fast-path results.
Reviewed by Cursor Bugbot for commit 80b7e5d. Configure here.
There was a problem hiding this comment.
Intentional, and now documented in the function header. The fast path defines the list as "traces whose start is in [since, until]" — step 2 on otel_traces_summary is authoritative for membership (the start filter) and ordering; step 1 is just a candidate generator (over-fetched 5x in 0d7b2ab so boundary drops never shrink the page). For the default list (until = now) an included trace started at/after since, so all its spans are in the window and the result equals the raw scan; the only divergence is a trace that started before since with spans reaching into the window — excluded here by design (it did not start in the window), versus the raw scan which includes it with a window-clipped start. For the short traces this serves that boundary sliver is negligible; validated top-100 identical to the raw ground truth on real data.
| ) | ||
| GROUP BY trace_id | ||
| ORDER BY min(ts) DESC | ||
| LIMIT {limit:UInt32} |
There was a problem hiding this comment.
Recent scan cap skews ranking
Medium Severity
Step 1 takes the newest TRACE_RECENT_SCAN_CAP spans, then ranks traces by min(ts) within that subset. That is not the same as ordering by true trace start in the window, so recently started traces can be missing or mis-ranked when older traces emit many late spans.
Reviewed by Cursor Bugbot for commit 80b7e5d. Configure here.
There was a problem hiding this comment.
Ordering is now authoritative from the summary, not the bounded step-1 scan: the final ORDER BY start_time DESC in step 2 ranks by each trace’s global min(start) from otel_traces_summary, and step 1 is only a candidate generator (over-fetched to 5x the page limit in 0d7b2ab). So a trace mis-ranked by step 1’s windowed min(ts) is re-ranked correctly by its true start in step 2, as long as it is in the candidate set — which the over-fetch ensures for the top page. Verified against the raw ground truth on a 7.8M-trace/24h project: top-100 trace_ids match 100/100 with zero field differences.
…p authoritative Address review feedback on the trace-list fast path: - Pre-backfill safety: the MVs only populate the derived tables from creation forward, so activating the fast path purely on table existence could return a truncated (post-migration-only) list where the raw scan showed the full window. Gate on the recent index actually holding a row older than the window start for the project; otherwise fall back to the raw scan. Activation is now self-correcting per project/window and no longer depends on deploy ordering. - Membership + ordering: step 1 (recent index) is only a candidate generator; step 2 (summary) is authoritative for "started in the window" and for the order. Over-fetch candidates (5x the page limit) so traces step 1 surfaces by recent activity but step 2 drops (start before the window) don't shrink the page below the requested limit. - Document the semantic: the list is "traces whose start is in [since, until]" with whole-trace stats, ordered by start. For the default list (until = now) every included trace's spans lie in the window, so stats equal the raw window-clipped values; they differ only for a historical window a long trace straddles.
1 participant
Problem
The explore traces grouped list groups raw
otel_tracesbyTraceIdover the whole selected window, then sorts and limits.otel_tracesis sorted by(ServiceName, SpanName, Timestamp)and the project filter lives in theResourceAttributesmap, so a project's list can't use the primary index — it scans the entire multi-tenant window. On a busy deployment a 24h range reads hundreds of millions of rows and exceeds the ClickHouse request timeout, so the list 500s (the user seesLOADING…→ error) while the rollup-backed chart above it still renders.A single per-trace rollup isn't enough on its own: a high-volume project can have millions of traces in a day, so "the 100 most recent" is still a
GROUP BY+ sort over all of them (~15s at 24h). The trace start can't be the sort key either — it's a min-aggregate, andSimpleAggregateFunctioncolumns aren't allowed in a key.Approach
Two derived tables (migration
005), read in two steps — the shape trace backends (Tempo/Jaeger) use:otel_traces_recent— a plain, time-ordered span index(project_id, ts, trace_id),ORDER BY (project_id, ts). A bounded reverse scan of the newest spans, grouped bytrace_id, yields the N most recently started trace_ids in milliseconds. Covers all traces (no root-span dependency).otel_traces_summary— oneAggregatingMergeTreerow per(project_id, trace_id)with start, span/error counts, root span/service/status (earliest span byTimestamp), distinct-service count, and the[start,end]nanos for total duration. Looked up for just those N trace_ids (IN, indexed) for the displayed stats.queryTracesAggregatedtakes this path only for the unfiltered list (the default view that times out); any span-selecting filter (service / span name / status code / attribute) or a duration floor falls back to the raw scan. Availability is probed once per client and memoized, so environments without the tables (local dev, or before the migration lands) fall back automatically.Validation
Measured against a real high-volume project (7.8M traces / 24h):
Rollout
The materialized views only capture spans ingested after creation. Historical traces need the one-shot backfill (
apps/worker/scripts/backfill-otel-traces-summary.ts), run over[retention_start, mv_cutover)before deploying the read path, so it never overlaps the live MV window (an overlap re-inserts spans and double-countsspan_count/error_count). Order: apply migration → backfill → deploy.Notes for review
otel_exceptions).otel_traces_recentis a full-span-volume index, so retention is worth a follow-up decision.TRACE_RECENT_SCAN_CAP = 50000bounds the recent-index reverse scan.Test plan
pnpm --filter @superlog/api test(32/32, incl. 4 new fast-path/fallback cases)pnpm --filter @superlog/api typecheck,pnpm --filter @superlog/worker typecheckNote
Medium Risk
Changes the production trace-list query path and introduces new derived tables whose correctness depends on migration order and a one-time backfill window; mis-ordered rollout or overlapping backfill can truncate lists or inflate aggregate counts, though filtered queries and missing rollups still fall back to raw scans.
Overview
Adds ClickHouse
otel_traces_recent(time-ordered span index) andotel_traces_summary(per-trace aggregates) with MVs, plus migration005and HA schema updates.queryTracesAggregatednow uses a two-step fast path (recent_ids→ summary*Mergestats) for the unfiltered default list when both tables exist and a coverage probe shows history reaches before the window start; otherwise it keeps the rawotel_tracesscan. Span/service/status/attribute filters andminDurationMsalways use raw. Table existence checks are generalized via memoizedEXISTS TABLE(replacing rollup-only probing).A one-shot
backfill-otel-traces-summary.tsfills pre-MV history in chunks (dry-run by default); rollout must avoid overlapping the live MV window to prevent double-counting in the summary.Tests add five cases for fast path vs fallback (filters, missing tables, incomplete window coverage).
Reviewed by Cursor Bugbot for commit 0d7b2ab. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Serve the Explore trace list from derived ClickHouse rollups, now gated by window coverage to avoid truncated results. The default unfiltered view loads in ~1s and matches the raw query, with automatic fallback when rollups are missing or incomplete.
New Features
otel_traces_recent(time-ordered span index) andotel_traces_summary(per-trace aggregates) with MVs; two-step read where the summary is authoritative for membership and order.queryTracesAggregateduses the fast path only when no filters/minDuration, both tables exist, and the recent index reaches before the window start; else falls back tootel_traces.limit * 5) to keep pages full; bounds the reverse scan withTRACE_RECENT_SCAN_CAP = 50000.tableExistsby client/table; used for both trace rollups andevents_per_minute.Migration
infra/clickhouse/migrations/005_otel_traces_summary.sql.apps/worker/scripts/backfill-otel-traces-summary.tsover[retention_start, mv_cutover); run once and avoid overlapping the live MV window.Written for commit 0d7b2ab. Summary will update on new commits.