Retry transient pool-acquire timeouts with exponential backoff

[swlynch99]

Summary

Add resilience to transient database connection pool timeouts by implementing automatic retry logic with exponential backoff instead of immediately tearing down the worker.

Key Changes

• New SharedState methods: Added acquire() and begin() methods that wrap the underlying pool operations with retry logic for transient PoolTimedOut errors
• Retry mechanism: Implemented PoolRetry struct that tracks exponential backoff state and decides whether to retry or propagate errors
  • Retries only on sqlx::Error::PoolTimedOut errors
  • Respects a configurable maximum retry count
  • Uses exponential backoff (doubling each time) with a configurable maximum backoff duration
  • Remains responsive to shutdown signals during backoff periods
• Configuration: Added three new config parameters to Config:
  • pool_acquire_max_retries: Maximum number of retries (default: 5)
  • pool_acquire_backoff: Initial backoff duration (default: 1 second)
  • pool_acquire_max_backoff: Maximum backoff duration (default: 30 seconds)
• Updated call sites: Replaced all direct pool.acquire() and pool.begin() calls throughout the worker with the new resilient methods
• Tests: Added unit tests for the next_backoff() helper function to verify correct doubling behavior and overflow handling

Implementation Details

• The retry logic is transparent to callers—they use the same API but get automatic resilience
• Transient pool timeouts (momentary spikes in connection demand) are now handled gracefully without worker teardown
• Sustained timeouts still escalate to worker teardown as before
• Shutdown signals interrupt backoff waits to prevent delays during graceful shutdown
• The backoff calculation uses saturating arithmetic to prevent overflow panics

https://claude.ai/code/session_01D4z4t3D4r4pTAYcXCSK8Ex