fix: faster MongoDB retry with reduced timeouts#31
Conversation
The API crashes immediately if MongoDB isn't reachable during startup (ServerSelectionTimeoutError). In Kubernetes, pods can start before MongoDB is fully available. Changes: - Retry loop: 10 attempts with 3s delay between retries - Reduced timeouts: serverSelection=5s, connect=5s, socket=10s (down from 30s/20s/20s defaults) so each retry fails fast - Each attempt now takes ~8s instead of ~33s - Structured logging for retry visibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR reduces MongoDB connection latency during startup failures by adding retry logic to the API’s Beanie/PyMongo initialization and lowering PyMongo timeouts so each retry attempt fails faster.
Changes:
- Add retry loop around
AsyncMongoClient+init_beanieinitialization. - Reduce PyMongo selection/connect/socket timeouts to speed up failed attempts.
- Add structured logging for retry attempts and final failure.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| client = AsyncMongoClient( | ||
| settings.mongo_uri, | ||
| serverSelectionTimeoutMS=5000, | ||
| connectTimeoutMS=5000, | ||
| socketTimeoutMS=10000, | ||
| ) | ||
| db = client[settings.mongo_db] | ||
| await init_beanie(database=db, document_models=ALL_MODELS) |
There was a problem hiding this comment.
client is assigned to the global before init_beanie(...) succeeds. This exposes a partially-initialized connection to other code (e.g., readiness checks importing app.database.client) and makes it harder to reason about whether client implies “DB ready”. Use a local variable for each attempt and only set the global client after Beanie initialization succeeds; close the local client on failure.
| except Exception as exc: | ||
| if attempt == MAX_RETRIES: |
There was a problem hiding this comment.
The retry loop catches Exception broadly. This will also retry on non-transient failures (e.g., invalid Mongo URI/auth config, model/import errors inside init_beanie), delaying startup and producing misleading “Database not ready” logs. Prefer catching only expected connection-related exceptions (e.g., pymongo.errors.PyMongoError / ServerSelectionTimeoutError) and let unexpected exceptions fail fast.
| logger.error( | ||
| "Database connection failed after all retries", | ||
| attempts=MAX_RETRIES, | ||
| error=str(exc), |
There was a problem hiding this comment.
On the final failure path, the log records only error=str(exc) and loses the traceback. For operational debugging, log the exception with stack trace (e.g., logger.exception(...) or logger.error(..., exc_info=True) depending on structlog config) before re-raising.
| error=str(exc), | |
| error=str(exc), | |
| exc_info=True, |
| MAX_RETRIES = 10 | ||
| RETRY_DELAY = 3 # seconds | ||
|
|
||
|
|
||
| async def init_db() -> None: | ||
| """Initialize MongoDB connection and Beanie ODM.""" | ||
| """Initialize MongoDB connection and Beanie ODM with retry logic.""" | ||
| global client | ||
| client = AsyncMongoClient(settings.mongo_uri) | ||
| db = client[settings.mongo_db] | ||
|
|
||
| from app.models import ALL_MODELS | ||
|
|
||
| await init_beanie(database=db, document_models=ALL_MODELS) | ||
| for attempt in range(1, MAX_RETRIES + 1): | ||
| try: | ||
| client = AsyncMongoClient( | ||
| settings.mongo_uri, | ||
| serverSelectionTimeoutMS=5000, | ||
| connectTimeoutMS=5000, | ||
| socketTimeoutMS=10000, | ||
| ) |
There was a problem hiding this comment.
MAX_RETRIES, RETRY_DELAY, and the timeout values are hard-coded. If this needs tuning per environment (local dev vs. CI vs. prod), consider moving these to Settings (env-configurable) so behavior can be adjusted without a redeploy/code change.
Problem
Each retry attempt waits 30s (PyMongo default \serverSelectionTimeoutMS) before failing, making total startup time 10 x 33s = 330s.
Root Cause
MongoDB pod is crashed due to volume permission issue (\VolumePermissionChangeInProgress). The Bitnami MongoDB container stopped but K8s shows \Running\ — all probes fail with \cannot exec in a stopped container.
MongoDB fix (run manually):
\\�ash
kubectl delete pod -n mongodb mongodb-557ff6d6c-r8tk9
\
If it keeps happening, add to MongoDB Helm values:
\\yaml
podSecurityContext:
fsGroupChangePolicy: OnRootMismatch
\\
API-side fix
Reduced connection timeouts so retries cycle faster:
Checklist