Skip to content

fix: faster MongoDB retry with reduced timeouts #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
atulmgupta wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: faster MongoDB retry with reduced timeouts #31

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 45 additions & 5 deletions api/app/database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,62 @@
"""MongoDB connection management using Beanie ODM."""

import asyncio

import structlog
from beanie import init_beanie
from pymongo import AsyncMongoClient

from app.config import settings

logger = structlog.get_logger()

client: AsyncMongoClient | None = None

MAX_RETRIES = 10
RETRY_DELAY = 3 # seconds


async def init_db() -> None:
"""Initialize MongoDB connection and Beanie ODM."""
"""Initialize MongoDB connection and Beanie ODM with retry logic."""
global client
client = AsyncMongoClient(settings.mongo_uri)
db = client[settings.mongo_db]

from app.models import ALL_MODELS

await init_beanie(database=db, document_models=ALL_MODELS)
for attempt in range(1, MAX_RETRIES + 1):
try:
client = AsyncMongoClient(
settings.mongo_uri,
serverSelectionTimeoutMS=5000,
connectTimeoutMS=5000,
socketTimeoutMS=10000,
)
Comment on lines +15 to +31

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAX_RETRIES, RETRY_DELAY, and the timeout values are hard-coded. If this needs tuning per environment (local dev vs. CI vs. prod), consider moving these to Settings (env-configurable) so behavior can be adjusted without a redeploy/code change.

Copilot uses AI. Check for mistakes.
db = client[settings.mongo_db]
await init_beanie(database=db, document_models=ALL_MODELS)
Comment on lines +26 to +33

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

client is assigned to the global before init_beanie(...) succeeds. This exposes a partially-initialized connection to other code (e.g., readiness checks importing app.database.client) and makes it harder to reason about whether client implies “DB ready”. Use a local variable for each attempt and only set the global client after Beanie initialization succeeds; close the local client on failure.

Copilot uses AI. Check for mistakes.
if attempt > 1:
logger.info("Database connected after retry", attempt=attempt)
return
except Exception as exc:
if attempt == MAX_RETRIES:
Comment on lines +37 to +38

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry loop catches Exception broadly. This will also retry on non-transient failures (e.g., invalid Mongo URI/auth config, model/import errors inside init_beanie), delaying startup and producing misleading “Database not ready” logs. Prefer catching only expected connection-related exceptions (e.g., pymongo.errors.PyMongoError / ServerSelectionTimeoutError) and let unexpected exceptions fail fast.

Copilot uses AI. Check for mistakes.
logger.error(
"Database connection failed after all retries",
attempts=MAX_RETRIES,
error=str(exc),

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the final failure path, the log records only error=str(exc) and loses the traceback. For operational debugging, log the exception with stack trace (e.g., logger.exception(...) or logger.error(..., exc_info=True) depending on structlog config) before re-raising.

Suggested change
error=str(exc),
error=str(exc),
exc_info=True,

Copilot uses AI. Check for mistakes.
)
raise
logger.warning(
"Database not ready, retrying...",
attempt=attempt,
max_retries=MAX_RETRIES,
delay=RETRY_DELAY,
error=str(exc),
)
# Close failed client before retrying
if client:
try:
await client.close()
except Exception:
pass
client = None
await asyncio.sleep(RETRY_DELAY)


async def close_db() -> None:
Expand Down