feat: add neon-drill, loading restored backups into Neon branches

[leochong]

Why

restore-test proves a backup restores, then deletes the cluster it timed. Nobody can query the result, so "the bytes came back" is as far as the evidence goes.

neon-drill runs the same rehearsal and leaves the recovered data in a throwaway Neon branch — cheap, isolated, and reachable by an ordinary connection string, so a person or a test suite can go and look.

The constraint that shaped the design

A wal-g physical backup cannot be restored into Neon. Neon keeps pages in its own pageserver, so there is no data directory to write and no replication protocol to stream into. Neither backup-fetch nor backup-push can address it — local backup-push asserts wal-g runs on the same host and calls pg_backup_start; remote mode needs REPLICATION for BASE_BACKUP; and both first call GetPgServerInfo, which needs pg_control_system(). Neon grants none of it.

So the drill is two-stage by necessity:

S3/GCS ──backup-fetch──> scratch PGDATA ──pg_ctl──> cluster ──pg_dump|psql──> Neon branch

The physical restore remains the thing under test. The branch is what it leaves behind.

The Neon load is deliberately outside the RTO verdict. It is a transfer bounded by dump-and-load throughput; judging it against a recovery budget would fail the drill for reasons unrelated to backup health. rto covers fetch and replay; the load is reported as neon_load_seconds.

What's here

• pkg/neon — control-plane client: branch create/list/delete, connection URI, async operation polling. Sits outside pkg/storages and is not registered in StorageAdapters — Neon cannot hold backups, and registering it as a backend would imply it can.
• internal/databases/postgres/neon_drill.go — reuses the existing drill machinery (ValidateDrillTarget, spacePhase, runFetchPhase, writeRecoveryConfig, stopDrillCluster, the DrillPhase/DoctorStatus vocabulary) rather than duplicating it. NeonDrillOptions/NeonDrillReport embed their restore-drill counterparts, so the restore half reports identically to a plain drill.
• cmd/pg/neon_drill.go, cmd/pg/neon_branches.go — the latter carries NoStorage, since it never reads backup storage.
• WALG_NEON_* config keys; the API key joins secretSettings.
• Docs in PostgreSQL.md, BACKUP-RECOVERY.md, README.md, COPYRIGHT.md.

Which database gets dumped

The drill does not guess. Once the restored cluster is running it queries pg_database and picks the single connectable, non-template database that is not postgres. The ambiguous cases are explicit:

• one user database — chosen silently and named in the neon-load detail;
• several — the drill stops and lists them. A Neon branch holds one database, so picking for you could load the wrong data under a green verdict;
• none but postgres — proceeds, but warns rather than passing, since it has almost certainly moved an empty database.

An explicit --source-database / WALG_NEON_SOURCE_DATABASE is never second-guessed and skips the query, so the drill still works when the catalog cannot be read.

The catalog query builds an explicit DSN rather than going through Connect(), which falls back to localhost:5432 and would otherwise be able to inspect the live cluster instead of the restored one.

Safety

A branch is a billable compute endpoint, so:

• neon-auth runs first — bad credentials fail in seconds, not after an hour of restoring a backup that has nowhere to go
• the branch is deleted on every exit path, SIGINT/SIGTERM included
• only branches carrying the walg-drill- prefix are ever deleted, so a bug in cleanup cannot destroy a hand-made branch
• the branch password reaches psql through the environment, never argv
• the API key is scrubbed from control-plane error text. A test pins this — it failed first time, because the error echoed the body verbatim, so an endpoint reflecting the Authorization header would have put the key straight into a CI log

Shared-code change

runReplayPhase gains a leaveRunning bool. A plain drill stops the restored cluster immediately; the Neon drill dumps from it next. One call site updated.

Verification

• pkg/neon tests use httptest (operation polling, list filtering, delete refusals, 404-as-success, rate limiting, key-never-leaks).
• Drill tests use a fake client: auth failure creates no branch, load failure still deletes the branch, --keep-branch retains, failed delete reports fail, plus the source-database selection cases.
• Full sweep over ./cmd/... ./internal/... ./pkg/... ./utility/... is green except TestInterpretTypeSymlink and TestHandlePITRWindow_MinWindowGate, both confirmed pre-existing by running them at a7e17e9b in a scratch worktree.
• Lint clean for these files. Two misspell hits are suppressed with //nolint: cancelling/cancelled are Neon's wire values, and Americanising them would stop them matching.
• Ran the built binary against a stub control plane — config resolution, auth, branch filtering, JSON output and the 401 path all behave.

A test also caught a real parser bug: packaged builds print pg_dump (PostgreSQL) 15.6 (Debian 15.6-1.pgdg120+2), and taking the last field yielded (Debian).

Not covered by tests: the pg_database query and the dump/load pipe both need a live cluster, so they are exercised only by the integration path — same as the existing queryRecoveryPoint.

Scope / follow-ups

Per the agreed scope this is the create-and-load side plus listing. Deferred: data-correctness verification (row counts/checksums) and wiring into doctor / backup-verify / retention-validate / pitr-window.

🤖 Generated with Claude Code