Skip to content

feat(drive): Google Drive folder sync — backend (Phase 10)#96

Merged
nbkdoesntknowcoding merged 2 commits into
nbkdoesntknowcoding:mainfrom
Jason-jo17:feat/drive-backend
Jul 13, 2026
Merged

feat(drive): Google Drive folder sync — backend (Phase 10)#96
nbkdoesntknowcoding merged 2 commits into
nbkdoesntknowcoding:mainfrom
Jason-jo17:feat/drive-backend

Conversation

@Jason-jo17

@Jason-jo17 Jason-jo17 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Design / tracking issue: #99 — part of the Google Drive folder-sync feature.

What & why

Bring-your-own-OAuth Google Drive integration — part 1 of 3 (backend). Modelled on the existing Google Calendar link (lib/google-calendar.ts, secret-box, the per-member refresh-token column) but wired to core tables (folders/docs/attachments), so verify-core-only stays green. Adds schema + idempotent migration 0073 (drive_refresh_token + drive_folder_links + drive_file_mappings), lib/google-drive.ts (OAuth + Drive REST via googleapis, configurable scope), lib/drive-sync.ts (pull text→docs / binary→attachments+R2, push docs→.md, idempotent by md5/hash), routes/drive.ts (connect/callback/status/folders/links CRUD/sync/webhook), a BullMQ queue + worker, and optional GOOGLE_DRIVE_* env (routes 503 until configured).

Linked issue

No single issue — see below.

Stacked: merge this first — the frontend + docs PRs build on it. New dep googleapis was discussed/approved (Drive changes + resumable-upload surface).

Test evidence

@boppl/api typechecks clean (tsc --noEmit: 0 errors) and builds. Schema ↔ migration 0073 verified column-for-column (17/17, 12/12, + the token column) — no table mismatch. Live OAuth + sync needs an operator’s Google Cloud OAuth app + running stack (steps in docs/connect/drive.md, shipped in the docs PR); the @boppl/api test suite needs Postgres+Redis and runs in CI.

Checklist

  • One logical change; matches the surrounding code style.
  • Ran the relevant checks for what I touched (see Test evidence).
  • Commits are signed off for the DCO (git commit -s).
  • This is a core change — not enterprise-adjacent and not in the licensing/release machinery.
  • I understand this PR is merged here on GitHub once approved (authorship + DCO preserved) and ships in the next tagged release.

@CLAassistant

CLAassistant commented Jul 13, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Bring-your-own-OAuth Google Drive integration, modelled on the existing
Google Calendar link (lib/google-calendar.ts, secret-box, the
workspace_members refresh-token column) but wired to core tables
(folders/docs/attachments) — no enterprise coupling, so verify-core-only
stays green.

- schema + migration 0073: workspace_members.drive_refresh_token (encrypted)
  plus drive_folder_links (Mnema⇄Drive folder pairing, direction, accepted
  types, conflict policy) and drive_file_mappings (per-file idempotency +
  conflict tracking). Idempotent SQL, applied by the psql migrate service.
- lib/google-drive.ts: OAuth (offline refresh token) + Drive REST via the
  googleapis SDK; configurable scope (drive.file default, opt-in drive).
- lib/drive-sync.ts: pull (Drive→Mnema: text→docs via markdownToYjsState,
  binary→attachments+R2) and push (Mnema→Drive: docs→.md), keyed on
  md5/content-hash so re-runs never duplicate; manual conflict policy never
  clobbers un-pushed local edits.
- routes/drive.ts: connect/callback/status, folder picker, links CRUD,
  "sync now", conflicts, and a Google push webhook (auth-exempted, matched
  by stored channel id).
- queue/drive-sync.ts + workers/drive-sync worker (registered in
  workers/server.ts); route registered in server.ts.
- env: GOOGLE_DRIVE_CLIENT_ID/SECRET/REDIRECT_URI, GOOGLE_DRIVE_SCOPE,
  DRIVE_SYNC_MAX_FILE_MB, DRIVE_WEBHOOK_SECRET — all optional; routes 503
  until configured.
- dep: googleapis (discussed — Drive changes/resumable-upload surface).

@boppl/api typechecks clean. Live end-to-end needs an operator's Google
Cloud OAuth app + running stack (see docs/connect/drive.md).

CI: registers migration 0073 in the drizzle journal so CI's `db:migrate`
applies it (the psql migrate service already applies it on self-host) — without
this the new `workspace_members.drive_refresh_token` column is absent in the CI
DB and core inserts fail. Adds a `url-template -> BSD-3-Clause` license-gate
override (transitive dep of googleapis; bundled LICENSE is 3-clause BSD, but
package.json reports the legacy non-SPDX "BSD").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Jason-jo17 <jason@theboringpeople.in>

@nbkdoesntknowcoding nbkdoesntknowcoding left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really solid backend — thank you. The security model is right: it mirrors our existing Google Calendar integration (AES-GCM-encrypted per-member refresh tokens, HMAC-signed OAuth state, editor-gated + workspace-scoped routes), and it correctly avoids SSRF by going through the googleapis SDK rather than fetching arbitrary URLs. That clears the bar we set after the OnlyOffice fix.

Two things to resolve before we merge:

  1. The /api/drive/webhook endpoint is currently dead code — and it's public + unauthenticated. Nothing in the PR ever registers a Drive watch channel, so drive_channel_id is never written; the webhook's channel-id lookup can never match and it always 204s. It also never verifies DRIVE_WEBHOOK_SECRET (that env var is defined but unused). We don't want to ship a public unauthenticated endpoint that looks security-relevant but does nothing. Please either:

finish it — register the changes.watch channel on link creation, persist drive_channel_id, and verify the incoming DRIVE_WEBHOOK_SECRET/x-goog-* headers; or
drop it for this PR (remove the route, the DRIVE_WEBHOOK_SECRET env var, and the channel column) and keep the working poll/manual-sync path. We can add push-sync as a follow-up.
2. Please confirm the googleapis dependency is intended. It adds ~30 transitive packages to the API bundle. We're fine with it if it's deliberate — just want an explicit yes rather than it slipping in.

Smaller nits (non-blocking, fix if easy): the per-file sync loops swallow errors silently (catch { skipped++ }) — a req.log.warn there would help debugging; and attachments.format gets raw extensions (png, csv) where the rest of the code expects docx/pdf — worth normalizing.

Addresses review on nbkdoesntknowcoding#96:
- Remove the /api/drive/webhook route, its public auth-plugin exemption,
  the DRIVE_WEBHOOK_SECRET env var + driveWebhookSecret() helper, and the
  drive_channel_id / drive_channel_expires_at / drive_page_token columns
  (schema + migration 0073). None of it was wired (no changes.watch was
  ever registered), so it was dead, public, unauthenticated code. The
  working manual/poll sync path stays; Drive push-sync can be a follow-up.
  Also drops the now-unused changes-API helpers (listChanges /
  getStartPageToken / getFileParents).
- Normalize attachments.format: binaries are limited to docx/pdf (the
  formats the attachment pipeline models); other accepted types are skipped
  with a log rather than stored with a foreign extension. Default accepted
  types are now text + docx/pdf.
- Log swallowed per-file sync errors (console.warn) for debuggability.

@boppl/api typechecks clean; schema <-> migration 0073 match column-for-column.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Jason-jo17 <jason@theboringpeople.in>
Jason-jo17 added a commit to Jason-jo17/mnema that referenced this pull request Jul 13, 2026
…ents

Follows the nbkdoesntknowcoding#96 rework: removes the DRIVE_WEBHOOK_SECRET env row and the
Google-push-notification/webhook mention (that path was dropped as
unfinished; live push-sync is a follow-up), and scopes the "what syncs"
copy to text -> docs and docx/pdf -> attachments (the v1 supported types).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Jason-jo17 <jason@theboringpeople.in>
@Jason-jo17

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review — addressed in 03bc143.

  1. Dropped the webhook. Removed the /api/drive/webhook route, its public auth-plugin exemption, the DRIVE_WEBHOOK_SECRET env var + driveWebhookSecret() helper, and the drive_channel_id / drive_channel_expires_at columns (plus the now-unused drive_page_token) from the schema and migration 0073. Also removed the dead changes-API helpers (listChanges / getStartPageToken / getFileParents). The working manual/poll sync path stays; live Drive push-sync is a clean follow-up.
  2. googleapis — yes, deliberate. It's the intended dependency for the Drive changes + resumable-upload surface (vs. hand-rolling REST like the calendar client). Explicit 👍.

Nits: per-file sync errors now console.warn with link/file context; attachments.format is normalized — binaries are limited to docx/pdf (the formats the attachment pipeline models), other accepted types are skipped with a log, and the default accepted types are now text + docx/pdf.

@boppl/api typechecks clean and schema ↔ migration 0073 match column-for-column. Docs PR #98 is updated to match (webhook refs removed).

@nbkdoesntknowcoding nbkdoesntknowcoding merged commit 11cb484 into nbkdoesntknowcoding:main Jul 13, 2026
5 checks passed
nbkdoesntknowcoding pushed a commit that referenced this pull request Jul 13, 2026
…ents (#100)

Follows the #96 rework: removes the DRIVE_WEBHOOK_SECRET env row and the
Google-push-notification/webhook mention (that path was dropped as
unfinished; live push-sync is a follow-up), and scopes the "what syncs"
copy to text -> docs and docx/pdf -> attachments (the v1 supported types).

Signed-off-by: Jason-jo17 <jason@theboringpeople.in>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants