Phase 3: human-in-the-loop at scale, secrets rotation, reputation monitoring#9
Merged
Conversation
…itoring
Completes the code-buildable remainder of Phase 3 (production
readiness); docs/phase3-readiness.md records what code provides vs the
operator/legal deliverables and the three parked decisions.
Human-in-the-loop at scale:
- The review queue is confidence-ordered (highest fit_score first,
FIFO within a score) and carries fit_score, so reviewers batch the
confident tail and spend attention on the uncertain bottom.
- POST /outreach-drafts/batch-review: up to 100 rubric decisions per
call through the same domain review path, each item in its own
transaction - one stale/bogus item fails alone, the rest land.
Never sends, like every review surface.
- Edit rate (approved_with_edits share of window reviews) is a
first-class metric: edits-as-signal for prompt iteration.
Secrets rotation:
- POST /internal/tenants/{id}/rotate-key (admin token): issues a new
tenant API key, old key dies at commit, rotation audited, key
returned exactly once.
- RELAY_MASTER_KEY_PREVIOUS: verify-only rotation window for the
master key, so signed unsubscribe tokens already sitting in
delivered mail keep working across a rotation - a dead unsubscribe
link is a compliance failure. New tokens always sign with the
current key.
Reputation monitoring:
- 24h suppressions-by-reason and reviews-by-decision in tenant
metrics; bounce_rate / complaint_rate / edit_rate properties wired
through /metrics and the Prometheus export.
- bounce_rate_high critical alert (RELAY_ALERT_BOUNCE_RATE, default
5%) with a min-sends floor (RELAY_ALERT_BOUNCE_RATE_MIN_SENDS) so a
1-of-1 bounce never pages - it fires BEFORE the eligibility
threshold silently pauses sending.
Tests (+7, suite now 299): batch review item isolation and terminal
states, confidence ordering, key rotation kills the old key instantly
(and 422/403/404 edges), master-key rotation keeps old tokens alive,
reputation + edit-signal metrics end to end, bounce alert honors the
min-sends floor then fires critical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASAVj4XgJCH3UcHkZZaYzM
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughThis PR adds three Phase 3 production-readiness capabilities: confidence-ordered batch human review of outreach drafts, tenant/master API key rotation with backward-compatible unsubscribe token verification, and bounce-rate reputation monitoring with alerting. Metrics, schemas, configuration, documentation, and tests are extended accordingly. ChangesPhase 3 Production Readiness
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Completes the code-buildable remainder of Phase 3 — Production Readiness (on the phase's own branch, per the new per-phase branching rule). With pacing/warmup and one-click unsubscribe already merged, the roadmap's remaining code items were: human-in-the-loop at scale, secrets rotation, and reputation monitoring. What code cannot close — production sending posture, legal/DPA documents, the human security review, and the three parked decisions — is now recorded explicitly in
docs/phase3-readiness.md.What changed
Human-in-the-loop at scale
/outreach-drafts/pendingsorts byfit_score(highest first, FIFO within a score) and carries the score — the top of the queue is the batchable tail, the bottom is where reviewer attention belongs.POST /outreach-drafts/batch-reviewtakes up to 100 rubric decisions per call through the same domain review path as the single endpoint. Each item runs in its own transaction, so one stale or bogus item fails alone while the rest land. Like every review surface, it never sends.approved_with_editsshare of window reviews is now a first-classedit_ratemetric — the number that steers prompt iteration.Secrets rotation
POST /internal/tenants/{id}/rotate-key(admin token): issues a new tenant API key, the old key dies the moment the rotation commits (rotation is for suspected exposure — a grace overlap would defeat the point), the action is audited, and the key is returned exactly once.RELAY_MASTER_KEY_PREVIOUS: a verify-only rotation window for the master key. Unsubscribe tokens already sitting in delivered mail keep verifying across a rotation — a dead unsubscribe link is a compliance failure. New tokens always sign with the current key.Reputation monitoring
bounce_rate,complaint_rate, andedit_rate— wired through/metricsJSON and the Prometheus export.bounce_rate_highcritical alert (RELAY_ALERT_BOUNCE_RATE, default 5%) with a min-sends floor (RELAY_ALERT_BOUNCE_RATE_MIN_SENDS, default 5) so a 1-of-1 bounce never pages. It fires before the eligibility threshold silently pauses sending.docs/phase3-readiness.md(new)Maps each exit-gate concern to its mechanism and tests; lists the operator/legal deliverables code must not invent (production posture per the §6 revisit, region-specific lawful-basis rules pending the legal artifact, DPA/incident-response, KMS swap, human review); records the three parked decisions (email-hash pepper, global-suppression asymmetry,
sequence_stepgeneralization).Tests (+7, suite now 299)
sent: false.Verification
ruff check+ format clean..env.example; no default-behavior change except the two new metrics fields (additive) and the alert (needs ≥5 sends/24h to ever fire).🤖 Generated with Claude Code
https://claude.ai/code/session_01ASAVj4XgJCH3UcHkZZaYzM
Generated by Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation