fix(sync): discover newly archived threads

[obviyus]

Summary

• discover active and newly archived bot-visible threads during routine latest-only syncs
• persist per-parent public/private archive cursors and stop pagination at the last successful scan
• fully index new or incomplete thread histories while retaining the one-page bootstrap for ordinary cursorless channels
• skip impossible private-archive requests for forum and announcement parents

Root cause

Steady-state catalog sync called only Discord's guild-active-threads endpoint. A thread created and archived between runs disappeared from that endpoint before Discrawl learned its channel ID, so its messages were never indexed.

Tests

• regression before fix: TestSyncLatestOnlyDiscoversAndCompletesNewArchivedThread failed with expected 1 thread, actual 0
• GOWORK=off go test ./...
• make lint
• make fmt
• autoreview: clean, no accepted/actionable findings

Live proof

Branch-scoped publish-discord-backup run 32280457716 passed at exact head 654f930cb8215bc1064e44ea5eee1f09bf8198e9.

• restored the latest main DB cache; no cold snapshot import
• archived-thread catalog scan completed in about 54 seconds
• persisted 200 archived-thread endpoint cursors; 47 accessible endpoints advanced to this scan, while 153 inaccessible endpoints retained their prior cursor for retry
• observed and published one real thread transition from active to archived between snapshots
• message sync processed 822 channels in 29 seconds, wrote 9,437 messages, and deferred zero channels
• 37 previously incomplete thread histories reached history_complete
• published archive commit: dc5fd83ad0977f3b3735304b1451d0633bd002e8

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs changes before merge. Reviewed August 19, 2026, 1:41 PM ET / 17:41 UTC.

ClawSweeper review

What this changes

The PR updates Discrawl’s routine Discord sync to discover newly archived threads, track per-parent archive cursors, and fully index new thread histories.

Merge readiness

⛔ Blocked by patch quality or review findings - 4 items remain

Keep open: the live backup evidence supports the new discovery path, but the prior P1 finding remains on the unchanged head—archive cursors advance before the newly discovered threads are durably synchronized, so a later failure can permanently skip their messages.

Priority: P1
Reviewed head: 654f930cb8215bc1064e44ea5eee1f09bf8198e9

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The production backup evidence is strong, but the unresolved P1 retry-boundary defect makes the patch not ready to merge.
Proof confidence	🦞 diamond lobster (5/6) ✨ media proof bonus	Sufficient (linked_artifact): The linked branch-scoped Discord backup run at the exact head reports a real archived-thread transition and successful after-fix indexing; redact any private server details in future proof updates.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (linked_artifact): The linked branch-scoped Discord backup run at the exact head reports a real archived-thread transition and successful after-fix indexing; redact any private server details in future proof updates.
Evidence reviewed	7 items	Cursor is committed too early: The new per-parent cursor is persisted immediately after the archive endpoint succeeds, before catalog persistence, member refresh, and message synchronization. Later failures abort the sync after the cursor write: The caller performs channel storage, member refresh, and message synchronization after building the catalog, returning an error from any of those stages. Existing regression covers only the success path: The added archived-thread test confirms a successful scan and later no-op sync, but does not force a downstream failure and verify that the same archived thread is rediscovered.
Findings	1 actionable finding	[P1] Defer archive cursor advancement until downstream sync succeeds
Security	None	None.

Live Verification

Command: go run ./cmd/discrawl sync --help

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

Assertions:

• FAIL expect_output: Usage: discrawl sync

How this fits together

Discrawl builds a Discord channel catalog before storing channel records and synchronizing message histories into its local archive. The new archived-thread scan feeds that catalog, so its cursor determines which archived threads later syncs can still discover.

flowchart LR
A[Discord archived-thread API] --> B[Incremental catalog scan]
B --> C[New thread catalog]
C --> D[Store channels and sync histories]
D --> E{Guild sync succeeds?}
E -->|yes| F[Persist archive cursor]
E -->|no| G[Retry from prior cursor]

Loading

Before merge

• Defer archive cursor advancement until downstream sync succeeds (P1) - The cursor is written immediately after the archive request succeeds, but channel storage, member refresh, and message synchronization happen afterward. If any later stage fails, the next latest-only run starts after this cursor and never rediscovers the archived thread that was not durably indexed. The prior review raised this same blocker and this code is unchanged.
• Resolve merge risk (P1) - A catalog, member-refresh, or message-sync failure after cursor persistence makes the next routine sync request only newer archives, leaving the already discovered thread absent from the incremental catalog and its messages unindexed.
• Complete next step (P2) - The remaining P1 blocker is a concrete cursor-ordering repair with a clear failure-and-retry validation path.
• Improve patch quality - Defer archive-cursor persistence until downstream sync work succeeds and add a forced-failure retry regression test.

Findings

• [P1] Defer archive cursor advancement until downstream sync succeeds — internal/syncer/channel_catalog.go:244

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Change surface	11 files; production +115/-14, tests +211/-18, docs +6/-4	The change adds persistent cursor behavior and substantial regression coverage across the Discord client and sync pipeline.

Merge-risk options

Maintainer options:

1. Commit cursors after durable synchronization (recommended)
   Move archive-cursor persistence behind successful downstream catalog and message work, then add a retry regression test before merge.

Technical review

Best possible solution:

Persist each archive cursor only after the corresponding guild’s catalog storage and message synchronization succeed, with a failure-then-retry regression test proving the archived thread is rediscovered.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible with high confidence: let archive discovery succeed, then fail channel storage, member refresh, or message synchronization; the persisted cursor prevents the next latest-only run from rediscovering that thread.

Is this the best way to solve the issue?

No; the discovery approach is sound, but committing its cursor before downstream durable work completes is not a safe retry boundary.

Full review comments:

• [P1] Defer archive cursor advancement until downstream sync succeeds — internal/syncer/channel_catalog.go:244
  The cursor is written immediately after the archive request succeeds, but channel storage, member refresh, and message synchronization happen afterward. If any later stage fails, the next latest-only run starts after this cursor and never rediscovers the archived thread that was not durably indexed. The prior review raised this same blocker and this code is unchanged.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 5a0dc6f611f8.

Labels

Label changes:

• add P1: An interrupted routine sync can cause real archived-thread messages to be skipped until an operator runs a broader repair.
• add proof: sufficient: Contributor real behavior proof is sufficient. The linked branch-scoped Discord backup run at the exact head reports a real archived-thread transition and successful after-fix indexing; redact any private server details in future proof updates.
• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦞 diamond lobster and patch quality is 🦪 silver shellfish.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (linked_artifact): The linked branch-scoped Discord backup run at the exact head reports a real archived-thread transition and successful after-fix indexing; redact any private server details in future proof updates.
• remove P2: Current review triage priority is P1, so this older priority label is no longer current.
• remove rating: 🧂 unranked krab: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.
• remove status: 📣 needs proof: Current PR status label is status: ⏳ waiting on author.

Label justifications:

• P1: An interrupted routine sync can cause real archived-thread messages to be skipped until an operator runs a broader repair.
• merge-risk: 🚨 message-delivery: The new cursor durability ordering can suppress rediscovery of an archived thread whose messages were not successfully indexed.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦞 diamond lobster and patch quality is 🦪 silver shellfish.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (linked_artifact): The linked branch-scoped Discord backup run at the exact head reports a real archived-thread transition and successful after-fix indexing; redact any private server details in future proof updates.
• proof: sufficient: Contributor real behavior proof is sufficient. The linked branch-scoped Discord backup run at the exact head reports a real archived-thread transition and successful after-fix indexing; redact any private server details in future proof updates.

Evidence

Acceptance criteria:

• [P1] GOWORK=off go test ./internal/syncer ./internal/discord.
• [P1] make lint.

What I checked:

• Cursor is committed too early: The new per-parent cursor is persisted immediately after the archive endpoint succeeds, before catalog persistence, member refresh, and message synchronization. (internal/syncer/channel_catalog.go:244, 654f930cb821)
• Later failures abort the sync after the cursor write: The caller performs channel storage, member refresh, and message synchronization after building the catalog, returning an error from any of those stages. (internal/syncer/syncer.go:229, 654f930cb821)
• Existing regression covers only the success path: The added archived-thread test confirms a successful scan and later no-op sync, but does not force a downstream failure and verify that the same archived thread is rediscovered. (internal/syncer/syncer_test.go:832, 654f930cb821)
• Prior blocker remains unchanged: The prior ClawSweeper cycle identified the same cursor-durability concern; the current head is still 654f930 and the relevant file is unchanged from that reviewed SHA. (internal/syncer/channel_catalog.go:244, 654f930cb821)
• Real production-path proof: The PR body links a branch-scoped Discord backup run at the exact head and reports an observed active-to-archived transition, cursor advancement, and completed thread histories. (654f930cb821)
• Current-main boundary: The incremental archived-thread catalog implementation is introduced by this branch relative to current main 5a0dc6f, so the central behavior is not already implemented there. (internal/syncer/channel_catalog.go:222, 654f930cb821)

Likely related people:

• obviyus: Authored the current-main latest-only synchronization work that this branch extends. (role: recent current-main sync contributor; confidence: high; commits: 5a0dc6f611f8; files: internal/syncer/channel_catalog.go, internal/syncer/message_sync.go)
• steipete: History shows repeated ownership of latest-only and thread-catalog synchronization behavior. (role: historical sync-area contributor; confidence: high; commits: e1dcd1ea4902, be137b9a3495; files: internal/syncer/channel_catalog.go, internal/syncer/message_sync.go)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)

• reviewed 2026-08-19T17:17:28.188Z sha 654f930 :: needs real behavior proof before merge. :: [P1] Advance archive cursors only after durable sync work

[obviyus]

Landed via squash onto main.

• Validation: GOWORK=off go test ./...; make lint; make fmt
• Live proof: manual sync run 32280457716 passed; 200 archive cursors, 9,437 messages indexed, 37 thread histories completed, 0 deferred failures
• Changelog: no manual entry; operator docs updated and release notes are release-owned
• Land commit: 654f930cb8215bc1064e44ea5eee1f09bf8198e9
• Merge commit: 938b6b279dfc5c0d534664386c5132d44c77ea1d