Bug 2053278 - Github-ETL export is hitting rate limits for importing data into bigquery tables and needs to be fixed

[dklawren]

What I changed (main.py)

1. Retry-with-backoff on all BigQuery writes

• Added _retry_bigquery_write() plus a _is_retryable_bq_error() predicate, mirroring the existing GitHub backoff style.
• It retries the transient conditions — rateLimitExceeded (your exact error), backendError, internalError, and 429/5xx — with jittered exponential backoff, so parallel workers that all back off don't retry in lockstep.
• It deliberately does not retry quotaExceeded (the hard 1,500-ops/day ceiling), since retrying a daily quota just burns time and fails anyway.
• Wired into the three write sites: the load job in _insert_rows_to_table, the carry-forward INSERT…SELECT, and the reconcile DELETE.

2. Batched the full-export load path

• Instead of one load job per 100-PR chunk, process_repo's full-export branch now accumulates transformed rows and flushes every _FULL_EXPORT_FLUSH_PRS PRs (default 1000). For a large repo this cuts load jobs ~10x.
• I refined my original proposal ("load once per repo") to a bounded flush so it doesn't regress the memory-efficiency the architecture is built around — the flush threshold caps peak memory while still collapsing the op count. Extraction/transform stay streamed per-chunk.
• Each flush hands off a fresh dict rather than mutating the one passed to load_data (avoids a mutate-after-handoff footgun).

Verification

• Full test suite: 144 passed. I updated 2 tests in test_main.py that asserted the old one-load-per-chunk behavior (now assert the batched single load) and confirmed the data-inspection tests still see the correct rows.
• ruff and black clean on both changed files. (The 2 test_formatting failures in this sandbox are environment-only: an older black lacking a py314 target and ruff not on PATH — plus unrelated lint in your untracked scratch file verify_incremental_emulator.py.)

Tuning knobs for the k8s deployment

• GITHUB_ETL_MAX_WORKERS — lower it (e.g. 2–3) to reduce concurrent writes hitting the shared tables.
• The new _FULL_EXPORT_FLUSH_PRS (1000) and _BQ_WRITE_MAX_RETRIES/_BQ_WRITE_BASE_DELAY constants are easy to adjust if you still see pressure.

To confirm the fix against real quotas, the INFORMATION_SCHEMA.JOBS query I gave earlier will show write-op counts per table per minute before and after.

[Copilot]

Pull request overview

This PR addresses BigQuery table-modification rate limiting during GitHub-ETL exports by reducing the number of write operations and adding retry-with-backoff for transient BigQuery write failures, improving reliability when multiple repos run in parallel and write to shared tables.

Changes:

• Added jittered exponential-backoff retry logic for BigQuery load jobs and DML writes.
• Batched full-export loads by accumulating transformed rows across chunks and flushing at a configurable PR threshold.
• Updated tests to reflect the new “accumulate then flush” full-export behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
main.py	Adds BigQuery write retry/backoff and batches full-export loads to reduce write-op bursts.
tests/test_main.py	Updates assertions to match the new full-export batching/flush behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[shtrom]

r+ w/nits