Data Proxy synchronises BigQuery tables to a PostgreSQL database (pg_duckdb) and exposes them through a PostgREST REST API, enforcing row-level security against a backend-managed access policy table, keyed by JWT identity claims.
BigQuery is the authoritative data store. PostgreSQL is a disposable, eventually consistent read cache.
- Architecture — service objectives, pipeline components, standalone and HA diagrams.
- Backups — encrypted backups of
access_policytables, and how to restore one. - Development — running the full pipeline on Kubernetes with Minikube and Helm.
- Environment Variables — configuration read by the producer, worker, and finalizer.
- Helm Chart — installing and configuring the chart, including HA mode.
- Security — creating users and row-level security.
- Sync — the
sync.jsonfile, table fields, and strategy selection. - Using the API — querying tables through PostgREST: filtering, ordering, pagination, counting.
This project is licensed under the Apache License 2.0. See LICENSE for the full text.