fix(api): stop per-request MCP session leak; cap glibc arenas (#96)

[mruwnik]

Summary

Fixes the primary root cause of #96: the api container's RSS ratcheting from ~264MB to its 1GiB limit over ~5 days.

• Upgrade mcp 1.10.0 → 1.22.0 — mcp ≤1.11 never calls http_transport.terminate() in _handle_stateless_request, so every POST /mcp leaks a live anyio task + ServerSession + memory streams for the life of the process (~1,150/day in prod). The upstream fix (terminate after handle_request) landed in mcp 1.12.0; verified present in 1.22.0.
• Pin fastmcp==2.13.3 (was floating >=2.10.0) — newest 2.x line that avoids fastmcp 2.14's runtime fakeredis<2.35 dependency (conflicts with our dev pin fakeredis==2.36.0) and pins a narrow compatible mcp range. Pinned exactly since fastmcp minors break APIs; the requirements comment says to bump both together.
• MALLOC_ARENA_MAX=2 on the api and worker containers — freed transient search/ingest working sets otherwise scatter across up to 32 glibc arenas and never return to the OS, pinning RSS at its historical peak (root cause 2 in the issue).

Test fallout

fastmcp ≥2.13's get_access_token() converts foreign token types with claims=None, which its own model rejects. The test auth helpers (conftest.py mcp_auth_context + local copies in test_teams.py/test_meta.py) now build fastmcp's AccessToken — the same type verify_token returns in production — instead of the bare SDK one.

Verification

• MCP test suite: 1167 passed, 0 failed against the new pins (was 279 failures before the harness fix, all one root cause).
• Full non-qdrant suite: 5307 passed; the 12 remaining failures reproduce identically on the old pins (known sandbox qdrant gap).
• pip install --dry-run resolves .[api] and .[all] cleanly; redis stays 5.3.1.
• Installed mcp 1.22.0 _handle_stateless_request confirmed to terminate the transport.

Post-deploy check (from the issue): RSS curve should flatten at ~300MB instead of creeping ~7MB/h; optionally verify len(asyncio.all_tasks()) stays in the dozens.

Remaining #96 items (payload-free candidate search, client caching, per-service metrics, MCP user_id attribution, oauth_client reaper) are intentionally out of scope.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TD6Ef83TfvdqxyczzCKpCA

[mruwnik]

Raven (Claude Opus 4.8): Reviewed. This is a clean, well-justified fix for #96 with no blocking issues. I independently verified the load-bearing claims rather than taking the description on faith:

Central leak fix — confirmed. Downloaded both wheels: mcp 1.10.0's _handle_stateless_request does NOT call http_transport.terminate(); mcp 1.22.0 DOES (terminate runs after handle_request, closing the streams and ending the anyio task). The root cause and the fix are real and present in the chosen version.

Dependency resolution — confirmed compatible. fastmcp 2.13.3 requires mcp!=1.21.1,<1.23,>=1.19.0; mcp==1.22.0 satisfies it. fastmcp 2.14 requires mcp>=1.23.1, so pinning to 2.13.3 to stay on the tested mcp range is sound.

Test-harness change — complete coverage. Exactly three sites construct AccessToken directly (conftest.py, test_meta.py, test_teams.py) and all three are switched to fastmcp's AccessToken (matching what verify_token returns in prod). The other 14 MCP test files build auth via conftest's mcp_auth_context, so the single conftest fix covers them. Nothing missed.

MALLOC_ARENA_MAX=2 — valid and well-targeted. Effective because all runtime images are python:3.12-slim (glibc, not musl). Placed on the api service and on the worker-env anchor, so it also reaches worker and ingest-hub — the services that actually hold large transient ingest payloads. environment overrides env_file, so it takes effect. Comments are accurate and appropriately explain the "why".

Minor nits (non-blocking):

1. The requirements-api.txt comment attributes 2.14 avoidance to a fakeredis conflict; that dep is actually only in 2.14.7, while the hard blocker across all of 2.14 is mcp>=1.23.1. Inline note left — reword only, decision is correct.
2. Upstream (not this PR's concern): 1.22.0's terminate() runs after handle_request but not in a finally, so a raising handle_request would still leak. Rare path; out of scope.

I can't run the MCP suite here (mcp/fastmcp aren't installed in the sandbox), so I'm relying on the PR's "1167 passed" for the runtime-behavior surface — but the production imports (FastMCP, get_access_token, middleware, mcp.types, auth provider/settings/shared.auth) are stable across this range and the test count is strong evidence. Approve.

[mruwnik]

Raven (Claude Opus 4.8): Re-review after CI went red. Request changes — the PR is not mergeable as-is: it turns a green pyright into 4 errors.

The backend check fails at the "Run linters" step. I traced it to pyright (ruff passes) and reproduced locally: pip install .[all] + mcp==1.22.0/fastmcp==2.13.3 yields 4 pyright errors, all in production files this PR doesn't modify. Downgrading to master's resolved pins (mcp 1.10.0 + fastmcp 2.11.2) gives 0 errors on the same files, so the dependency bump is the sole cause. Details and per-line fixes are in the inline comment on the mcp==1.22.0 line; summary:

• oauth_provider.py:944 load_access_token returns the SDK AccessToken but fastmcp 2.13's base now expects FastMCPAccessToken (incompatible override). Same class of mismatch you fixed in the test helpers — check whether it's also a latent runtime bug here, not just typing.
• oauth_provider.py:1020 RefreshToken(client_id=…) — client_id is now str | None.
• oauth_provider.py:610 iterating client_info.redirect_uris, now Optional.
• visibility_middleware.py:106 on_list_tools narrows CallNext[…, Sequence[Tool]] to list[Tool].

The rest of the PR is sound and I verified the substantive claims independently: the leak fix is real (mcp 1.10.0's _handle_stateless_request never calls terminate(); 1.22.0 does, right after handle_request); mcp==1.22.0 sits inside fastmcp 2.13.3's mcp>=1.19.0,<1.23 range; the three direct AccessToken construction sites are all updated and the other 14 MCP test files go through the fixed conftest helper; and MALLOC_ARENA_MAX=2 is effective (glibc python:3.12-slim) and reaches api + worker + ingest-hub.

So: approve the approach, but the production type-mismatches must be fixed (or CI's pyright made to pass) before merge. Per this repo's convention, red CI on the branch — even in files the PR didn't intend to touch — is in scope for the PR. Happy to re-review once the linter step is green.

[mruwnik]

Raven (Claude Opus 4.8): Approve. The new commit (430f678) fixes all 4 pyright regressions and CI is green (backend + frontend both pass). I verified the fixes independently, not just by reading: applied the head versions of oauth_provider.py and visibility_middleware.py and ran pyright + ruff under the new pins (mcp 1.22.0 / fastmcp 2.13.3) — 0 errors, 0 warnings on both.

Each fix is correct and minimal:

• load_access_token now returns Optional[FastMCPAccessToken] and both return sites construct FastMCPAccessToken, matching fastmcp 2.13's base signature (and consistent with verify_token). The now-unused SDK AccessToken import was dropped.
• load_refresh_token guards client.client_id is None up front — pyright narrows the later RefreshToken(client_id=…) to str, and the early return is semantically right (a client with no ID has no tokens).
• redirect_uris or [] handles the now-Optional field.
• on_list_tools widened to Sequence[Tool] to match the base CallNext contract; the body still returns a list (a valid Sequence).

The whole PR now holds together: the leak fix (terminate() in mcp 1.22.0, independently confirmed), the fastmcp/mcp pin (in-range), the arena cap (glibc, reaches api + workers + ingest-hub), the test-harness AccessToken switch, and now the production type-signature updates the bump required.

One trivial, non-blocking leftover: the requirements-api.txt comment still says "2.14 requires fakeredis" — that dep is only in 2.14.7, and the real blocker for 2.14 is mcp>=1.23.1. Comment accuracy only; the pin is correct. No objection to merging as-is.