Fix hybrid content queries with multi-parent server-side index fan-out by MrDirkelz · Pull Request #1818 · bccsa/luminary

MrDirkelz · 2026-07-13T14:31:20Z

Split hybridQuery content requests using parentId: {$in: [...]} into individual parentId equality queries pinned to
content-parentId-publishDate-index, then merge, sort, and limit their results server-side. This avoids Mango’s full content-partition scan caused by combining $in with a publishDate sort.

Add query-service coverage for the fan-out behavior and document why clients may use the above-cap multi-parent query path now that the API can execute it efficiently.

MrDirkelz · 2026-07-16T07:28:49Z

can look at http3 later

Look at if a frield on contentDto can be used to improve query instead of batches

Leaving the HTTP/3 and contentDto-field ideas for a later pass — not addressing in this PR.

ivanslabbert

Remove / update inline comments to not describe functionality of imported functions, causing documentation to easily drift and become invalid

ivanslabbert · 2026-07-22T07:31:25Z

Extract fitting functions to utility functions to make the code a bit shorter and easily readable

Done — moved the free-standing helpers out of the class body:

mapWithConcurrency, findParentIdIn, applySortAndLimit, setBlockRange → api/src/util/queryFanout.ts

extractMemberOf, removeMemberOf, extractFieldFromAnd → api/src/util/querySelector.ts

query.service.ts is down to ~370 lines (was ~520). Same convention as the other flat api/src/util/*.ts files. Verified with tsc --noEmit, eslint, and the full query.service.spec.ts suite (42/42 passing, unchanged).

ivanslabbert · 2026-07-22T07:35:42Z

    const isBelowCutoffContent = (d: BaseDocumentDto): boolean => {
        if (d.type !== DocType.Content) return false;
        const content = d as ContentDto;
+        if (content.parentType !== DocType.Post) return false;


Check if needed. This might cause non-Post content to not be synced

Full reasoning on the sync.ts thread below (isWindowedContentSubType), but specifically on the "might cause non-Post content to not be synced" worry: this line actually does the opposite — it exempts Tag content from the below-cutoff gate entirely, so Tag docs are always persisted on a live update regardless of cutoff (that's the liveSync.spec.ts "persists below-cutoff Tag content without a retention row" test). Now reads !isWindowedContentSubType(content.parentType), same behavior, single source of truth with sync.ts/retention.ts.

ivanslabbert · 2026-07-22T07:35:59Z

+        it("tag content sync uses the open publishDate range without a companion run", async () => {
+            await sync({
+                type: DocType.Content,
+                subType: DocType.Tag,
+                memberOf: ["group1"],
+                languages: ["en"],
+                limit: 100,
+                includeDeleteCmds: false,
+            });
+
+            expect(syncBatch).toHaveBeenCalledTimes(1);
+            expect(syncBatch).toHaveBeenCalledWith(
+                expect.objectContaining({
+                    type: DocType.Content,
+                    subType: DocType.Tag,
+                    publishDateMin: OPEN_MIN,
+                    publishDateMax: OPEN_MAX,
+                }),
+            );
+            expect(vi.mocked(syncBatch).mock.calls[0][0].alwaysOffline).toBeUndefined();
+        });
+


Check above comment

Same reasoning as the sync.ts thread — no change needed here, this test still passes unchanged and continues to pin the behavior (Tag content sync stays open-ended, no companion always-offline run) that isWindowedContentSubType now centralizes.

ivanslabbert · 2026-07-22T07:37:06Z

+    // publishDate is a Content-only sync dimension. Post content uses the configured cutoff
+    // so sync does not pull ordinary posts older than the app/HybridQuery treats as
+    // "remote-only". Tag content must remain open-ended: tags can be long-lived navigation
+    // parents, so applying the rolling Post window can exclude every matching document.
+    // Non-Content callers (Language, Redirect, Storage, AuthProvider, Group, …) leave the
+    // bounds undefined; downstream comparisons resolve those as OPEN_MIN/MAX.
    if (options.type === DocType.Content) {
-        if (options.alwaysOffline) {
+        if (options.subType !== DocType.Post || options.alwaysOffline) {


Related to above comment. Remove workaround - bug needs to be fixed at root cause

Agreed the inline subType !== DocType.Post checks scattered across this file, liveSync.ts, and retention.ts were a workaround. Root-caused it: added a single canonical predicate, isWindowedContentSubType(subType), co-located with getContentPublishDateCutoff() in shared/src/config.ts. All four call sites (this gate, the companion-run gate above it, liveSync's isBelowCutoffContent, and retention's eviction filter) now call that one function instead of independently re-deriving the Post-vs-Tag policy inline — so it can't drift between files again.

I did consider deriving "windowed" from syncList state (per-column publishDateMin) instead of a static predicate, since that's closer to "ask the sync engine" rather than hardcoding a doctype. Traced it through and dropped it: content.parentType is already denormalized on the doc, so syncList lookups buy nothing, and it introduces a real race (a live push arriving before the first Post syncList column registers would read as "not windowed" and skip the cutoff gate). Went with the static single-source-of-truth version instead — happy to discuss if you had a different root cause in mind.

No behavior change — full shared test suite (127/127 in the touched files) passes unchanged.

ivanslabbert · 2026-07-22T07:37:33Z

+            const content = d as ContentDto;
+            return (
+                content.parentType === DocType.Post && content.parentAlwaysOffline !== true


related to above comment

Full reasoning on the sync.ts thread. Now reads isWindowedContentSubType(content.parentType) && content.parentAlwaysOffline !== true, same behavior, single source of truth with sync.ts/liveSync.ts.

MrDirkelz · 2026-07-22T12:47:50Z

@ivanslabbert re: your review summary comment "Remove / update inline comments to not describe functionality of imported functions, causing documentation to easily drift and become invalid" — this wasn't attached to a line so replying here.

Found the target: app/src/components/ExplorePage/PinnedTopics.vue, app/src/components/HomePage/HomePagePinned.vue, and app/src/components/VideoPage/PinnedVideo.vue each had a comment restating how the API's parentId fan-out works internally (per-parent index seeks, fan-out cap behavior) — that's implementation detail owned by queryIntrospection.ts/query.service.ts, exactly the kind of thing that drifts. Trimmed all three to state only what's relevant at the call site (why the older-tail supplement is required, why sort+limit are set) and pointed at queryIntrospection.ts as the source of truth instead of re-describing it.

- Extract query.service.ts's free-standing helpers into api/src/util/queryFanout.ts (parentId fan-out + result merging) and querySelector.ts (selector field extraction), shrinking the service file and matching the existing util/ layout. - Trim the duplicated fan-out comments in PinnedTopics/HomePagePinned/PinnedVideo that restated queryIntrospection.ts's internals instead of pointing to it. - Centralize the Post-vs-Tag publishDate-windowing check into a single isWindowedContentSubType() predicate in shared/config.ts, replacing four independent `=== DocType.Post` comparisons in sync.ts, liveSync.ts, and retention.ts so the policy can't drift between call sites. No behavior change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

MrDirkelz · 2026-07-27T13:38:16Z

Root cause — calcChunk/syncBatch path (fixed): The {$lte:0, $gte:0} full-table scan came from syncBatch issuing a /query for a column already at the epoch floor. calcChunk returns {blockStart: 0, blockEnd: 0} on the initialSync: false continuation when the column's only stored chunk has blockEnd === 0, and the existing sub-tolerance early-return only catches strictly inverted ranges (blockStart < blockEnd), so the equal both-zero range fell through to the POST.

Added a narrow early-return guard in shared/src/api/sync/syncBatch.ts that seals the column eof and skips the /query call when blockStart === 0 && blockEnd === 0. A healthy eof column's catch-up poll (blockStart = MAX_SAFE_INTEGER, blockEnd = frontier.blockStart - syncTolerance) is left intact so new-doc detection still works. The API's both-zero 400 guard (query.controller.ts) remains the server-side backstop.

400 cascade (verified safe — no change): A 400 → http.ts returns undefined → syncBatch throws → propagates to app/cms .catch → Sentry. No retry/backoff/re-queue exists anywhere in shared/src/api/sync/, and the sync iterators only bump on accessMap/isConnected/language changes — never on failure. A persistent 400 is a stuck-fail-fast column, not a retry storm. With the client guard, the both-zero query is never sent, so the backstop 400 effectively never fires for sync.

Verified: syncBatch.spec.ts 50/50 (2 new tests), full sync dir 303/303, npm run type-check passes. Changes confined to shared/src/api/sync/syncBatch.ts + spec — behavioural only, no cross-package contract touched.

Split `hybridQuery` content requests using `parentId: {$in: [...]}` into individual `parentId` equality queries pinned to `content-parentId-publishDate-index`, then merge, sort, and limit their results server-side. This avoids Mango’s full content-partition scan caused by combining `$in` with a `publishDate` sort. Add query-service coverage for the fan-out behavior and document why clients may use the above-cap multi-parent query path now that the API can execute it efficiently.

QueryService's per-parent fan-out (parentId.$in) had no upper bound, so a caller could force one CouchDB request per id with no cap — worse than the full-scan it replaced. Add a hard maxFanoutParents cap, bound concurrent CouchDB requests per fan-out, and feed the existing per-identity rate limiter immediately for oversized-but-allowed fan-outs rather than waiting on post-hoc query-cost stats. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

- Extract query.service.ts's free-standing helpers into api/src/util/queryFanout.ts (parentId fan-out + result merging) and querySelector.ts (selector field extraction), shrinking the service file and matching the existing util/ layout. - Trim the duplicated fan-out comments in PinnedTopics/HomePagePinned/PinnedVideo that restated queryIntrospection.ts's internals instead of pointing to it. - Centralize the Post-vs-Tag publishDate-windowing check into a single isWindowedContentSubType() predicate in shared/config.ts, replacing four independent `=== DocType.Post` comparisons in sync.ts, liveSync.ts, and retention.ts so the policy can't drift between call sites. No behavior change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

buildSyncContext/collectIncludedLanguages were added as temporary debug logging alongside the real updatedTimeUtc epoch-cursor fix and never cleaned up. Drop them and document the endpoint's actual request-time guards instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

calcChunk returns {blockStart: 0, blockEnd: 0} on the initialSync:false continuation when the column's only stored chunk has blockEnd === 0 (already at the epoch floor). The existing sub-tolerance early-return only catches strictly inverted ranges (blockStart < blockEnd), so the equal both-zero range fell through to the /query POST — producing the updatedTimeUtc $lte:0,$gte:0 selector that walked the whole CouchDB index returning no docs (#1815). Add a narrow early-return guard that seals the column eof and skips the /query call when blockStart === 0 && blockEnd === 0. A healthy eof column's catch-up poll (blockStart = MAX_SAFE_INTEGER, blockEnd = frontier.blockStart - syncTolerance) is left intact so new-doc detection still works. The API's both-zero 400 guard (query.controller.ts) remains the server-side backstop. Co-Authored-By: Claude <noreply@anthropic.com>

MrDirkelz self-assigned this Jul 13, 2026

MrDirkelz linked an issue Jul 13, 2026 that may be closed by this pull request

API, SHARED: Full table scan being done to couchDB unnesesarily #1815

Open

MrDirkelz commented Jul 16, 2026

View reviewed changes

MrDirkelz force-pushed the 1815-api-shared-full-table-scan-being-done-to-couchdb-unnesesarily branch from 5b6d4e3 to d33080b Compare July 21, 2026 09:11

ivanslabbert reviewed Jul 22, 2026

View reviewed changes

MrDirkelz force-pushed the 1815-api-shared-full-table-scan-being-done-to-couchdb-unnesesarily branch from 26755f9 to c0c27d6 Compare July 22, 2026 12:49

MrDirkelz and others added 7 commits July 27, 2026 15:48

Update query.service.ts

4b003a2

tmp

bedaa47

MrDirkelz force-pushed the 1815-api-shared-full-table-scan-being-done-to-couchdb-unnesesarily branch from 131a758 to 6d8fb8d Compare July 27, 2026 13:48

Uh oh!

Conversation

MrDirkelz commented Jul 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivanslabbert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MrDirkelz commented Jul 22, 2026

Uh oh!

MrDirkelz commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants