feat(drive): Google Drive folder sync — backend (Phase 10)#96
Conversation
Bring-your-own-OAuth Google Drive integration, modelled on the existing Google Calendar link (lib/google-calendar.ts, secret-box, the workspace_members refresh-token column) but wired to core tables (folders/docs/attachments) — no enterprise coupling, so verify-core-only stays green. - schema + migration 0073: workspace_members.drive_refresh_token (encrypted) plus drive_folder_links (Mnema⇄Drive folder pairing, direction, accepted types, conflict policy) and drive_file_mappings (per-file idempotency + conflict tracking). Idempotent SQL, applied by the psql migrate service. - lib/google-drive.ts: OAuth (offline refresh token) + Drive REST via the googleapis SDK; configurable scope (drive.file default, opt-in drive). - lib/drive-sync.ts: pull (Drive→Mnema: text→docs via markdownToYjsState, binary→attachments+R2) and push (Mnema→Drive: docs→.md), keyed on md5/content-hash so re-runs never duplicate; manual conflict policy never clobbers un-pushed local edits. - routes/drive.ts: connect/callback/status, folder picker, links CRUD, "sync now", conflicts, and a Google push webhook (auth-exempted, matched by stored channel id). - queue/drive-sync.ts + workers/drive-sync worker (registered in workers/server.ts); route registered in server.ts. - env: GOOGLE_DRIVE_CLIENT_ID/SECRET/REDIRECT_URI, GOOGLE_DRIVE_SCOPE, DRIVE_SYNC_MAX_FILE_MB, DRIVE_WEBHOOK_SECRET — all optional; routes 503 until configured. - dep: googleapis (discussed — Drive changes/resumable-upload surface). @boppl/api typechecks clean. Live end-to-end needs an operator's Google Cloud OAuth app + running stack (see docs/connect/drive.md). CI: registers migration 0073 in the drizzle journal so CI's `db:migrate` applies it (the psql migrate service already applies it on self-host) — without this the new `workspace_members.drive_refresh_token` column is absent in the CI DB and core inserts fail. Adds a `url-template -> BSD-3-Clause` license-gate override (transitive dep of googleapis; bundled LICENSE is 3-clause BSD, but package.json reports the legacy non-SPDX "BSD"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Jason-jo17 <jason@theboringpeople.in>
cdb9b3b to
f112f1f
Compare
nbkdoesntknowcoding
left a comment
There was a problem hiding this comment.
Really solid backend — thank you. The security model is right: it mirrors our existing Google Calendar integration (AES-GCM-encrypted per-member refresh tokens, HMAC-signed OAuth state, editor-gated + workspace-scoped routes), and it correctly avoids SSRF by going through the googleapis SDK rather than fetching arbitrary URLs. That clears the bar we set after the OnlyOffice fix.
Two things to resolve before we merge:
- The /api/drive/webhook endpoint is currently dead code — and it's public + unauthenticated. Nothing in the PR ever registers a Drive watch channel, so drive_channel_id is never written; the webhook's channel-id lookup can never match and it always 204s. It also never verifies DRIVE_WEBHOOK_SECRET (that env var is defined but unused). We don't want to ship a public unauthenticated endpoint that looks security-relevant but does nothing. Please either:
finish it — register the changes.watch channel on link creation, persist drive_channel_id, and verify the incoming DRIVE_WEBHOOK_SECRET/x-goog-* headers; or
drop it for this PR (remove the route, the DRIVE_WEBHOOK_SECRET env var, and the channel column) and keep the working poll/manual-sync path. We can add push-sync as a follow-up.
2. Please confirm the googleapis dependency is intended. It adds ~30 transitive packages to the API bundle. We're fine with it if it's deliberate — just want an explicit yes rather than it slipping in.
Smaller nits (non-blocking, fix if easy): the per-file sync loops swallow errors silently (catch { skipped++ }) — a req.log.warn there would help debugging; and attachments.format gets raw extensions (png, csv) where the rest of the code expects docx/pdf — worth normalizing.
Addresses review on nbkdoesntknowcoding#96: - Remove the /api/drive/webhook route, its public auth-plugin exemption, the DRIVE_WEBHOOK_SECRET env var + driveWebhookSecret() helper, and the drive_channel_id / drive_channel_expires_at / drive_page_token columns (schema + migration 0073). None of it was wired (no changes.watch was ever registered), so it was dead, public, unauthenticated code. The working manual/poll sync path stays; Drive push-sync can be a follow-up. Also drops the now-unused changes-API helpers (listChanges / getStartPageToken / getFileParents). - Normalize attachments.format: binaries are limited to docx/pdf (the formats the attachment pipeline models); other accepted types are skipped with a log rather than stored with a foreign extension. Default accepted types are now text + docx/pdf. - Log swallowed per-file sync errors (console.warn) for debuggability. @boppl/api typechecks clean; schema <-> migration 0073 match column-for-column. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Jason-jo17 <jason@theboringpeople.in>
…ents Follows the nbkdoesntknowcoding#96 rework: removes the DRIVE_WEBHOOK_SECRET env row and the Google-push-notification/webhook mention (that path was dropped as unfinished; live push-sync is a follow-up), and scopes the "what syncs" copy to text -> docs and docx/pdf -> attachments (the v1 supported types). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Jason-jo17 <jason@theboringpeople.in>
|
Thanks for the thorough review — addressed in 03bc143.
Nits: per-file sync errors now
|
11cb484
into
nbkdoesntknowcoding:main
…ents (#100) Follows the #96 rework: removes the DRIVE_WEBHOOK_SECRET env row and the Google-push-notification/webhook mention (that path was dropped as unfinished; live push-sync is a follow-up), and scopes the "what syncs" copy to text -> docs and docx/pdf -> attachments (the v1 supported types). Signed-off-by: Jason-jo17 <jason@theboringpeople.in> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
What & why
Bring-your-own-OAuth Google Drive integration — part 1 of 3 (backend). Modelled on the existing Google Calendar link (
lib/google-calendar.ts, secret-box, the per-member refresh-token column) but wired to core tables (folders/docs/attachments), soverify-core-onlystays green. Adds schema + idempotent migration0073(drive_refresh_token+drive_folder_links+drive_file_mappings),lib/google-drive.ts(OAuth + Drive REST via googleapis, configurable scope),lib/drive-sync.ts(pull text→docs / binary→attachments+R2, push docs→.md, idempotent by md5/hash),routes/drive.ts(connect/callback/status/folders/links CRUD/sync/webhook), a BullMQ queue + worker, and optionalGOOGLE_DRIVE_*env (routes 503 until configured).Linked issue
No single issue — see below.
Test evidence
@boppl/apitypechecks clean (tsc --noEmit: 0 errors) and builds. Schema ↔ migration0073verified column-for-column (17/17, 12/12, + the token column) — no table mismatch. Live OAuth + sync needs an operator’s Google Cloud OAuth app + running stack (steps indocs/connect/drive.md, shipped in the docs PR); the@boppl/apitest suite needs Postgres+Redis and runs in CI.Checklist
git commit -s).