Phase 4 completion: per-tenant sending identity, worker concurrency, exit-gate ledger

[Aakash1337]

Why

Finishes the code-buildable remainder of Phase 4 — Productization & Scale. With onboarding, quotas/spend controls, and cost attribution merged (#11), the two remaining mechanical items were the per-tenant mailbox/domain isolation model and concurrency scaling of the worker. The new docs/phase4-readiness.md is the exit-gate ledger: what's pinned by tests versus what needs an operator decision.

What changed

Per-tenant sending identity (mailbox/domain ownership seam)

• tenants.sender_from_address: each client sends as its own provider-verified address; NULL falls back to the global RELAY_SES_FROM (so the sandbox pilot is untouched).
• The worker resolves the tenant's identity once per pass, fail-closed (an unreadable tenant row skips the tenant with a recorded error — mail is never sent under the wrong identity) and hands it to the sender; the sender protocol gains an optional sender_identity parameter. Onboarding accepts the address.
• tenants.sender_identity_verified attest (second commit): a tenant sending as its own address is blocked at eligibility (tenant_sender_identity_verified) until the operator attests the identity is provider-verified via POST /internal/tenants/{id}/attest-sender-identity (audited). Without this, a typo'd onboarding address would reach SES, get rejected, and terminally fail every approved lead.
• The send.executed audit event records the identity each send actually used.

Worker concurrency (throughput scaling)

• process_pending(concurrency=N) / relay-worker --concurrency N: tenants drain in parallel threads. Safe by construction — tenants are independent work streams (per-job transactions under RLS, FOR UPDATE SKIP LOCKED claims, per-tenant advisory-lock cap serialization).
• max_jobs remains the GLOBAL pass budget (second commit): shared across tenant streams via a thread-safe counter, so a tick is bounded the same with one tenant or fifty.
• Concurrency is clamped to 8 against the 15-connection default engine pool (oversubscription would block threads on checkout and silently skip tenants); the sender registry's lazy construction is locked (concurrent boto3 client creation isn't thread-safe and a crash there landed after the job claim).

Exit-gate ledger — docs/phase4-readiness.md

Exit-gate item	Status
New client onboarded without hand-editing config	Done, pinned
Two tenants simultaneously, verified data and sending isolation	Done, pinned (exact RLS row sets; per-tenant From observed at the provider)
Per-client cost and profitability visible	Done (GET /economics)
Target throughput sustained under concurrent load	Mechanism done and pinned; the target is the operator's — set a number, then benchmark (--concurrency)

Review

The second commit is the result of an adversarial multi-agent self-review of the first (8 finder angles, all findings verified then fixed): global pass budget, registry lock, concurrency clamp, fail-closed identity resolution, the identity-verification attest gate, audit of the identity used, plus stats-merge/onboarding-schema cleanups.

Tests (+5 across both commits, suite now 317)

• Sending isolation: a tenant with its own attested identity sends as it while another uses the global one, in one worker pass.
• Unverified identity is blocked before the provider, with the check named in the block reason.
• Attest endpoint flow: false at onboarding → 200 attest → 409 nothing-to-attest → 404 unknown tenant.
• The concurrent worker drains a mixed two-tenant queue in one pass with exact isolation afterwards.
• max_jobs=3 across 2 tenants × 2 jobs processes exactly 3 (global budget proven).

Verification

• Full suite: 317 passed against real Postgres; ruff clean.
• Pilot DB migrated in place (identity + attest columns verified present).

🤖 Generated with Claude Code

https://claude.ai/code/session_01ASAVj4XgJCH3UcHkZZaYzM

Summary by CodeRabbit

• New Features

  • Added per-tenant sending identity support, including onboarding, attestation, and visible verification status.
  • Introduced concurrent send processing across tenants with a shared per-pass job budget.
  • Added a readiness document summarizing Phase 4 exit criteria and remaining operational decisions.

• Bug Fixes

  • Blocked sends when a tenant’s sending identity is set but not verified.
  • Improved sender client setup to avoid duplicate initialization under concurrent load.

• Documentation

  • Updated the roadmap and readiness guidance to reflect the current production posture and next steps.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2ed72d45-e145-4d61-9a66-f745729dc915

📥 Commits

Reviewing files that changed from the base of the PR and between 8983e8c and 66f3d6d.

📒 Files selected for processing (14)

• README.md
• docs/phase4-readiness.md
• src/relay/api/routes.py
• src/relay/api/schemas.py
• src/relay/db/models.py
• src/relay/db/sql/001_schema_evolution.sql
• src/relay/domain/eligibility.py
• src/relay/senders/base.py
• src/relay/senders/registry.py
• src/relay/senders/ses.py
• src/relay/senders/simulated.py
• src/relay/workers/send_worker.py
• tests/test_phase1c_send.py
• tests/test_phase4_scale.py

---

📝 Walkthrough

Walkthrough

This PR adds per-tenant sending identity with provider attestation, an eligibility gate blocking unverified sends, sender contract changes (DirectSender, SES, Simulated) to accept sender_identity, thread-safe sender caching, and concurrent multi-tenant worker draining with a global job budget, plus supporting schema, API, tests, and documentation.

Changes

Sender identity and worker concurrency

Layer / File(s)	Summary
Tenant model and schema src/relay/db/models.py, src/relay/db/sql/001_schema_evolution.sql	Adds sender_from_address and sender_identity_verified columns to the Tenant model and migration.
Onboarding and attestation API src/relay/api/schemas.py, src/relay/api/routes.py	Extends onboarding schemas with sender identity fields, adds TenantSenderAttestResponse, updates onboarding route to persist/return the field, and adds an admin attestation endpoint.
Eligibility gate src/relay/domain/eligibility.py	Loads tenant row earlier and adds a check blocking real sends when the sender identity is unattested.
Sender contract and registry src/relay/senders/base.py, src/relay/senders/ses.py, src/relay/senders/simulated.py, src/relay/senders/registry.py	Adds optional sender_identity param across senders (SES uses it to override From address), and adds thread-safe double-checked locking to sender caching.
Concurrent worker draining src/relay/workers/send_worker.py	Adds WorkerStats.merge, rewrites process_pending for concurrent multi-tenant draining with a global job budget, resolves per-tenant identity, and adds --concurrency CLI flag.
Tests and docs tests/test_phase1c_send.py, tests/test_phase4_scale.py, docs/phase4-readiness.md, README.md	Adds tests for identity override, blocking, concurrency, and budget enforcement; adds readiness doc and updates README.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Admin
  participant Routes as routes.py
  participant DB as Tenant DB
  participant Worker as send_worker
  participant Sender as SESSender

  Admin->>Routes: onboard tenant (sender_from_address)
  Routes->>DB: create Tenant
  Admin->>Routes: attest-sender-identity
  Routes->>DB: set sender_identity_verified=true
  Routes-->>Admin: TenantSenderAttestResponse

  Worker->>DB: discover tenants with queued jobs
  Worker->>DB: resolve sender_identity per tenant
  Worker->>Sender: send(sender_identity)
  Sender-->>Worker: message_id
  Worker->>DB: record audit event with sender_identity

Loading

Possibly related PRs

• Aakash1337/Relay#4: Both PRs modify process_pending(...) in src/relay/workers/send_worker.py, overlapping in orphan recovery and pass-processing logic.
• Aakash1337/Relay#5: Both PRs modify the real-send eligibility gating in src/relay/domain/eligibility.py and extend the sender call signature, building on the same SES sender seam.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/relay-phase-4

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}