A cloud-native, event-driven platform for ingesting, processing, and storing application-level usage events.
EventFlow is designed as an engineering-first platform that demonstrates how modern software systems should be structured when long-term scalability, maintainability, and data-readiness are treated as first-class concerns. The system focuses on building the core infrastructure required to support analytics and ML use cases, rather than being a finished analytics product.
EventFlow addresses the common problem of event collection and management in modern applications. Many teams face difficulties with:
- Tight coupling between product logic and analytics logic
- Lack of standardization in event definitions
- Poor scalability when event volume increases
- Limited visibility into collected events
- Difficult migration to analytics platforms later
EventFlow solves this by providing:
- ✅ Clean and scalable event ingestion API
- ✅ Centralized event storage with strong structure
- ✅ Visibility into collected events
- ✅ Extensibility for future analytics and ML use cases
EventFlow follows an event-first architecture with clear separation of concerns:
Frontend (Next.js) → Backend API (NestJS) → Event Store (PostgreSQL)
↓
Consumer Groups → Event Processing
- Frontend Dashboard - Next.js web application for managing streams, viewing events, and monitoring consumers
- Backend API - NestJS REST API for event ingestion, querying, and management
- Event Store - PostgreSQL database with append-only event storage
- Consumer System - Consumer groups with offset tracking for reliable event consumption
- Processing Pipeline - Async event processing with retry logic and dead letter queue
- Next.js 14+ (App Router) with TypeScript
- React 18+ with Tailwind CSS
- Zustand for global state management
- React Query for server state
- React Hook Form + Zod for form validation
- Node.js with NestJS framework
- PostgreSQL for event storage
- Redis for caching and rate limiting
- Prisma for database access and migrations
- JWT for authentication
- OpenTelemetry for observability
- AWS (EC2/ECS, RDS, CloudWatch)
- Docker for containerization
- Terraform for Infrastructure as Code
- GitHub Actions for CI/CD
This project includes comprehensive documentation covering the entire design and planning process:
- Problem Statement - Why EventFlow exists
- Why EventFlow - Design motivation
- Target Users - Who this is for
- Production-Ready Implementation Plan - Step-by-step implementation guide
- Functional Requirements - What the system must do
- Non-Functional Requirements - Quality attributes
- Constraints & Assumptions - Project boundaries
- High-Level Architecture - System components
- Technical Stack Mapping - Technology choices
- Data & Event Model Design - Data structure
- API Design - API contracts
- Cloud Architecture & Deployment - AWS deployment
- Security Design - Security measures
- Backend Execution Design - Backend implementation plan
- Backend Detailed Module Design - Module breakdown
- Database Schema Design - Complete database schema
- Backend Request & Event Flow - Runtime behavior
- Frontend-Backend Interaction - Integration design
- Frontend Detailed Module Design - Frontend architecture
- Testing Strategy - Testing approach
- CI/CD Pipeline Design - Deployment automation
- Monitoring & Alerting Design - Observability strategy
- Node.js 18+
- PostgreSQL 15+
- Redis 7+
- Docker (optional, for containerized deployment)
- AWS Account (for cloud deployment)
Copy backend/.env.example to backend/.env and frontend/.env.example to frontend/.env.local, then set your values.
We use a simple branch model to keep main stable and integrate work in one place:
main— Stable/production branch. Only updated by merging fromdevelopwhen a set of work is complete (e.g. a phase or release).develop— Default working branch. Day-to-day development and feature integration happen here.- Feature branches — Create from
develop(e.g.feature/phase-2-auth). When the feature is done, merge intodeveloponly (not intomain).
Flow: feature/* → develop → (when a set of code is complete) → main.
This setup keeps main deployable while all new work is integrated and tested on develop.
- ✅ Event ingestion via REST API
- ✅ Event stream management
- ✅ Consumer groups with offset tracking
- ✅ Dead letter queue for failed events
- ✅ Web dashboard for event visualization
- ✅ API key authentication
- ✅ Role-based access control
- ✅ Rate limiting
- ✅ Structured logging and metrics
- Real-time analytics dashboards
- Stream processing (Kafka integration)
- Machine learning pipelines
- Advanced query capabilities
- Multi-region replication
EventFlow implements comprehensive security measures:
- Authentication: JWT tokens for users, API keys for producers
- Authorization: Role-based access control (RBAC)
- Encryption: Data encrypted at rest and in transit
- Input Validation: Schema validation for all events
- Rate Limiting: Per-API-key rate limits
- Audit Logging: All security events logged
See Security Design for details.
The project follows a comprehensive testing strategy:
- Unit Tests: 80%+ coverage target
- Integration Tests: All API endpoints tested
- E2E Tests: Critical user journeys
- Performance Tests: Load testing for event ingestion
- Security Tests: Authentication and authorization testing
See Testing Strategy for details.
EventFlow includes comprehensive observability:
- Metrics: Business and technical metrics via CloudWatch
- Logging: Structured JSON logs with request tracing
- Tracing: Distributed tracing with OpenTelemetry/X-Ray
- Alerting: Multi-level alerts (Critical, High, Medium, Low)
- Dashboards: Custom CloudWatch dashboards
See Monitoring & Alerting Design for details.
This is a personal project demonstrating production-ready system design. The documentation shows the complete thought process and architectural decisions.
When contributing, follow the branching workflow above: work on develop or a feature branch off develop, and merge to main only when a phase or release is complete.
This project demonstrates modern software engineering practices including:
- Event-driven architecture
- Cloud-native design
- Production-ready planning
- Comprehensive documentation
prathamesh.patil000009@gmail.com
Built with careful planning and attention to production-ready architecture. 🚀