Intelligent Notification Service

Production-ready distributed notification platform: REST intake → Kafka → intelligent routing (ML) → grounded personalization (RAG + GenAI) → dispatch, with idempotency, retries, DLQ, rate limiting, metrics, and graceful degradation. One docker compose up.

Architecture

Client → REST API (Spring Boot)
              │  persist event (status=RECEIVED), enforce idempotency
              ▼
          Kafka topic: notifications.inbound
              │
              ▼
      Processor / Consumer (Spring Boot)
   ┌──────────┴───────────────────────────────────────────────┐
   │ 1. per-user rate-limit check (token bucket, Redis)        │
   │ 2. ML SIDECAR  → {channel, send_window}     (800ms + fb)  │
   │ 3. RAG LAYER   → user context snippets      (1500ms + fb) │
   │ 4. GENAI LAYER → personalized message       (1500ms + fb) │
   │ 5. dispatch to channel adapter (email/SMS/push, mocked)   │
   │ 6. delivery record → PostgreSQL (status transitions)      │
   └───────────────────────────────────────────────────────────┘
        success            → DELIVERED
        transient failure  → notifications.retry (exp backoff + jitter)
        exhausted          → notifications.DLQ + dead_letters + FAILED

Status lifecycle: RECEIVED → ROUTED → GENERATED → DISPATCHED → DELIVERED (or → RETRYING → FAILED). Every transition is persisted and returned by the status API.

Service	Stack	Port (host)
core-backend	Java 21, Spring Boot 3.3, Kafka, Postgres, Redis, Flyway	8080
ml-sidecar	FastAPI, LightGBM → ONNX Runtime, scikit-learn	8001
rag-layer	FastAPI, LangChain, Weaviate, sentence-transformers	8002
genai-layer	FastAPI, LangChain, Claude primary / OpenAI fallback	8003
postgres / redis / kafka / weaviate	infra	5432 / 6379 / 9092 / 8081

Quick start

cp .env.example .env        # works as-is: MOCK_LLM=true needs no API keys
docker compose up -d --build
python rag-layer/seed_rag.py   # optional: seed grounded user history (needs rag deps locally)
make smoke                  # POST → DELIVERED + history + idempotency probe

ML models are trained deterministically inside the ml-sidecar image build, so no seed step is required for the stack to run. make seed retrains models and reseeds RAG from your host (pip install -r ml-sidecar/requirements.txt -r rag-layer/requirements.txt).

Real LLM output: set ANTHROPIC_API_KEY (and optionally OPENAI_API_KEY for fallback) in .env, set MOCK_LLM=false, docker compose up -d genai-layer.

API

• POST /api/v1/notifications — {user_id, notification_type, idempotency_key, payload?} → 202 {notification_id, status}
• GET /api/v1/notifications/{id} — status + full transition history
• GET /api/v1/notifications?user_id=&status=&page=&size= — filtered, paginated
• GET /api/v1/dlq — dead letters; POST /api/v1/dlq/{id}/replay — requeue (409 if already replayed)
• GET /actuator/health, GET /actuator/prometheus

Reliability semantics

• Idempotency: dedupe on idempotency_key (DB unique index is the source of truth; races resolve via constraint violation). Kafka publish only on first insert → duplicates never re-enter the pipeline. Verified by tests and make smoke.
• Retry: transient failures → notifications.retry with x-attempt/x-not-before headers, exponential backoff + jitter (RETRY_* env), max attempts → DLQ.
• Rate limiting: atomic Redis token bucket per (user_id, notification_type) (RATE_LIMIT_* env). Over-limit events are deferred, not dropped — a separate x-defers counter that never consumes retry attempts. Redis outage → limiter fails open (delivery is never blocked).
• DLQ: notifications.DLQ topic + dead_letters table + replay API. No silent drops.

Graceful degradation (runbook)

ML, RAG and GenAI are optional: each call has a per-call timeout (800/1500/1500 ms) and a fallback (default email/morning routing, empty context, template message). None of them appear in core-backend's depends_on — the core starts and delivers without them.

Verify it live:

make degrade      # stops all 3 sidecars, posts, requires DELIVERED, restarts them

Fallback usage is visible in transition details (routing model=fallback, generation model=fallback-template) and in the sidecar_fallbacks_total{client=…} metric.

Observability

• Metrics (/actuator/prometheus): notification_pipeline_duration (p50/p95/p99), notifications_processed_total{outcome}, notification_retries_total, notifications_dlq_total, rate_limit_defers_total, sidecar_fallbacks_total{client}.
• Correlation: every log line carries nid=<notification_id> (MDC), set by the inbound and retry consumers.

Testing

make test                                # everything below, phase-guarded
cd core-backend && ./gradlew test        # unit + Testcontainers E2E (Docker required)
cd ml-sidecar   && python -m pytest      # features, contract, cold-start, ONNX≈LightGBM
cd rag-layer    && python -m pytest      # contract + (with Weaviate up) live round-trip
cd genai-layer  && python -m pytest      # contract, guardrails, provider chain (MOCK_LLM)

Failure injection for manual testing: payload {"force_fail": true} (always fails → DLQ) or {"fail_times": N} (fails N dispatch attempts, then succeeds).

Environment variables

Every variable is documented inline in .env.example — LLM keys (optional; MOCK_LLM default true), Postgres/Redis/Kafka/Weaviate endpoints, embeddings provider, sidecar client flags + timeouts (ML_TIMEOUT_MS=800, RAG_TIMEOUT_MS/GENAI_TIMEOUT_MS=1500), rate-limit and retry knobs. Secrets live in .env only — never in code or images.

Definition of done — status

docker compose up all healthy ✓ · make smoke → DELIVERED with full history ✓ · duplicate keys never double-send ✓ · retry/DLQ/replay verified by tests ✓ · make degrade delivers with all sidecars down ✓ · tests green ✓ · PROGRESS.md final ✓ (items marked ✓ pending your local run — this repo was built in a sandbox without Docker; see PROGRESS.md Phase 6 notes for the exact verification commands).

Build history

Phase-by-phase commits; see PROGRESS.md for per-phase notes and decisions, and HANDOVER.md (project doc) for locked architectural decisions.