fix(sync): bootstrap cursorless channels in latest-only mode

[obviyus]

Problem

Routine and scheduled --latest-only syncs discover new active channels and threads, but skip their messages forever because those surfaces do not yet have a latest_message_id cursor. Channel metadata is published while the message archive remains empty.

Fix

• Fetch at most one newest page when latest-only sync encounters a cursorless channel.
• Persist the newest message cursor and oldest fetched backfill boundary.
• Leave longer history incomplete so a later --full run resumes from that boundary.
• Reuse the canonical backfill loop; no thread-specific behavior.
• Align the routine-sync documentation; leave release-owned changelog curation untouched.

Real Discord transport proof

Ran this branch with a real Discord bot against a brand-new disposable config and SQLite archive. Guild/channel IDs and credentials are redacted; no message content was inspected or retained.

$ discrawl sync --source discord --guild <redacted> --skip-members --latest-only
message sync started ... channels=6 full=false
message sync finished ... messages_written=261 ... elapsed=2s
{guilds:1, channels:14, threads:0, members:0, messages:261}

$ sqlite3 <disposable-db> <cursor-state-query>
kind=text stored_count=100 latest_cursor=set backfill_cursor=set history_complete=""

$ discrawl sync --source discord --guild <redacted> --channels <redacted> --full --since 2022-01-01T00:00:00Z --skip-members
message sync finished ... messages_written=34 ... elapsed=3s
kind=text stored_count=134 oldest_time=2022-01-17T15:06:21Z latest_cursor=set backfill_cursor=set history_complete=""

$ discrawl sync --source discord --guild <redacted> --channels <redacted> --full --skip-members
message sync finished ... messages_written=3578 ... elapsed=35s
kind=text stored_count=3712 oldest_time=2018-07-21T13:13:45Z latest_cursor=set backfill_cursor=set history_complete=1

This proves the production Discord client fetched one bounded head page for a cursorless channel, persisted both cursor boundaries, resumed older history from that boundary, and completed the archive on a later full pass. The disposable archive was deleted afterward.

Automated proof

• Regression test reproduces an active forum thread with 101 messages. Latest-only stores exactly 100 with one API call; full sync later stores the remaining message and marks history complete.
• GOWORK=off go test ./...
• make lint
• make test-race
• make test-coverage (85.1%)
• make smoke
• Autoreview: clean, no actionable findings.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 19, 2026, 10:43 AM ET / 14:43 UTC.

ClawSweeper review

What this changes

The PR makes latest-only Discord archive sync fetch one newest page for a newly discovered channel or thread without a cursor, then retain cursor state for a later full backfill.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

Keep open: current main still skips cursorless channels in latest-only mode, while this PR adds the bounded bootstrap and has corrected the previously missed guide. No source-level correctness blocker was found; wait for the in-progress test check before merge.

Priority: P2
Reviewed head: ff97f331b6bfa3a97cda819ee4c5068806d009bd

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	A focused implementation with direct production-transport evidence, stateful regression coverage, and no actionable review finding.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.
Evidence reviewed	6 items	Current-main defect: Current main returns before requesting messages whenever latest-only mode finds no stored latest cursor, which matches the reported cursorless-channel failure. Bounded implementation: The PR routes a cursorless latest-only channel through the existing backfill paginator with a one-page limit, then advances the latest cursor. Regression coverage: The new forum-thread regression verifies one 100-message page and one API call, preserves an incomplete backfill cursor, then completes the remaining history in a later full sync.
Findings	None	None.
Security	None	None.

Live Verification

Command: go run ./cmd/discrawl sync --help

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

Assertions:

• FAIL expect_output: --latest-only

How this fits together

Discord sync discovers accessible channels and threads, then writes messages and per-channel cursors to the local SQLite archive. Latest-only runs update the newest available data, while full runs continue from the stored backfill boundary.

flowchart LR
A[Discord channels and threads] --> B[Sync mode selection]
B --> C[Stored cursor check]
C --> D[One-page latest bootstrap]
C --> E[Incremental or full backfill]
D --> F[SQLite messages and cursors]
E --> F

Loading

Before merge

• Complete next step (P2) - No mechanical repair is identified on the current head; this PR awaits ordinary maintainer review and completion of its running test check.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Changed surface	4 files: 1 production, 1 test, 2 docs	The implementation remains focused on the message-sync path and its mirrored operator guidance.
LOC delta	production +26/-5; tests +48/-15; docs +2/-2	The production growth is limited to a bounded reuse of the existing backfill path and is covered by a stateful regression.

Technical review

Best possible solution:

Merge the bounded bootstrap after required checks complete so newly discovered active surfaces receive a current message page and a later full sync can finish their history.

Do we have a high-confidence way to reproduce the issue?

Yes—source-reproducible with high confidence: current main returns immediately for latest-only channels without a stored cursor, and the PR regression exercises the resulting forum-thread path.

Is this the best way to solve the issue?

Yes. Reusing the established backfill paginator with a one-page cap preserves cursor semantics and avoids a separate thread-specific synchronization path.

AGENTS.md: found, but no applicable review policy affected this item.

Codex review notes: model internal, reasoning high; reviewed against 679ed223ab13.

Labels

Label changes:

• add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

• P2: This repairs routine archive synchronization for newly discovered Discord channels and threads with a bounded user impact.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.
• proof: sufficient: Contributor real behavior proof is sufficient. Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.

Evidence

What I checked:

• Current-main defect: Current main returns before requesting messages whenever latest-only mode finds no stored latest cursor, which matches the reported cursorless-channel failure. (internal/syncer/message_sync.go:236, 679ed223ab13)
• Bounded implementation: The PR routes a cursorless latest-only channel through the existing backfill paginator with a one-page limit, then advances the latest cursor. (internal/syncer/message_sync.go:239, ff97f331b6bf)
• Regression coverage: The new forum-thread regression verifies one 100-message page and one API call, preserves an incomplete backfill cursor, then completes the remaining history in a later full sync. (internal/syncer/syncer_test.go:593, ff97f331b6bf)
• Prior review finding resolved: The matching public sync-sources guide now describes cursorless one-page bootstrap, removing the contradiction identified in the prior review. (docs/guides/sync-sources.md:21, ff97f331b6bf)
• Feature history: Latest-only mode originated in Peter Steinberger's commit, and Hannes Rudolph later hardened monotonic latest cursor handling in the same sync path. (internal/syncer/message_sync.go:222, e1dcd1ea4902)
• Real transport proof: The PR body supplies redacted terminal output from a real Discord bot and SQLite archive: a bounded latest-only fetch, persisted latest/backfill cursors, then partial and complete full backfills. (ff97f331b6bf)

Likely related people:

• Peter Steinberger: Introduced the original latest-only mode and remains the primary current-main history signal for this sync path. (role: introduced latest-only sync behavior; confidence: high; commits: e1dcd1ea4902, 089c1f5b6ae4; files: internal/syncer/message_sync.go, internal/syncer/syncer_test.go)
• Hannes Rudolph: Authored the monotonic latest-checkpoint repair in the same message synchronization code. (role: recent cursor-state contributor; confidence: high; commits: 265a76cb90e3; files: internal/syncer/message_sync.go)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)

• reviewed 2026-08-19T13:32:35.980Z sha 84fa9b7 :: needs real behavior proof before merge. :: [P3] Update the routine-sync documentation | [P3] Remove the release-owned changelog entry
• reviewed 2026-08-19T14:34:07.693Z sha c730a64 :: needs changes before merge. :: [P3] Update the matching sync-sources guide

[obviyus]

@clawsweeper automerge

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.