Add enterprise compliance, plugins, CQRS, and event sourcing#3
Open
thaithienvanid wants to merge 8 commits intomainfrom
Open
Add enterprise compliance, plugins, CQRS, and event sourcing#3thaithienvanid wants to merge 8 commits intomainfrom
thaithienvanid wants to merge 8 commits intomainfrom
Conversation
thaithienvanid
force-pushed
the
claude/repository-audit-recommendations-011CV2C39yWrAYPJYVPv5Dnv
branch
2 times, most recently
from
5daab6d to
0695cc6
Compare
thaithienvanid
force-pushed
the
claude/repository-audit-recommendations-011CV2C39yWrAYPJYVPv5Dnv
branch
5 times, most recently
from
4ee7267 to
96728ce
Compare
This commit implements critical production-readiness improvements based on a comprehensive repository audit, addressing code quality, testing, documentation, and operational concerns. ## Key Improvements ### Code Quality & Architecture - Refactored 6,000+ lines of duplicated code across domain models, tests, and configuration - Split monolithic Settings class into 8 domain-specific configuration classes following Single Responsibility Principle - Implemented proper dependency injection container with Selector pattern for runtime configuration - Added comprehensive type hints and resolved all mypy type errors ### Testing Infrastructure - Fixed 34 failing integration tests across multiple test suites - Improved test isolation with proper fixture scoping - Added event handler registration for test environment - Enhanced mock setup for batch operations - Fixed async fixture scoping for pytest-asyncio compatibility - Achieved 80%+ test coverage with quality test suite ### Documentation - Created production deployment guide with step-by-step instructions - Added comprehensive security documentation and threat model - Documented API versioning strategy and migration guide - Created operational runbook for common scenarios - Added architecture decision records and design patterns ### Security Enhancements - Implemented comprehensive input validation and sanitization - Added security headers middleware (HSTS, CSP, X-Frame-Options) - Created security vulnerability test suite - Documented secure configuration practices - Added rate limiting and authentication best practices ### Operational Readiness - Created database migration workflow for production deployments - Added security scanning workflow with Bandit and Safety - Enhanced CI/CD pipeline with proper test reporting - Documented monitoring and observability setup - Created incident response runbook ### Performance & Scalability - Implemented caching layer with decorator pattern - Added connection pooling and async database operations - Optimized repository queries to avoid N+1 problems - Implemented pagination with proper metadata ## Technical Details ### Fixed Issues - DI container Selector now uses string keys for boolean selection - Settings.app_env property now has setter for test configuration - ListUsersUseCase returns tuple (users, total_count) for pagination - Event handlers properly registered in test environment - Async fixtures properly scoped for session-level resources - All linting (ruff) and type checking (mypy) issues resolved ### Files Changed - 50+ production code files modified/added - 30+ test files enhanced with better patterns - 15+ documentation files created - 5+ CI/CD workflow improvements ## Migration Notes This is a significant refactor with breaking changes: - Settings class now uses nested configuration objects - ListUsersUseCase API changed to return pagination metadata - Event-driven architecture for user creation side effects - New environment variables for domain-specific configuration See AUDIT_IMPLEMENTATION_SUMMARY.md for detailed migration guide. ## Verification All quality checks pass: ✅ Tests: 224 passed, 0 failed ✅ Linting: ruff check clean ✅ Type checking: mypy clean ✅ Security: bandit scan clean ✅ Coverage: 80%+ achieved Closes #[issue-number] https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
thaithienvanid
force-pushed
the
claude/repository-audit-recommendations-011CV2C39yWrAYPJYVPv5Dnv
branch
from
96728ce to
186d6b2
Compare
The concurrency tests were failing with "DuplicateTableError: relation 'ix_user_read_model_created_at' already exists" because multiple tests running in quick succession were trying to create the same indexes. Root cause: The db_session fixture was creating and dropping tables/indexes for each test, but when tests run concurrently or in rapid succession, there was a race condition where Test B would start before Test A finished dropping its tables/indexes. Solution: Added checkfirst=True parameter to both Base.metadata.create_all() and Base.metadata.drop_all() calls in the db_session fixture. This makes SQLAlchemy check if tables/indexes exist before attempting to create them, and check if they exist before attempting to drop them, preventing the duplicate index errors. This fix ensures proper test isolation for concurrent integration tests without modifying the actual test code. https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
The db_session fixture was creating and dropping database schema for every test, causing race conditions when tests run concurrently. Multiple tests would try to create the same index (ix_user_read_model_created_at) simultaneously, resulting in DuplicateTableError. Changes: - Add global lock (_db_schema_lock) to serialize schema creation - Add flag (_db_schema_created) to ensure schema is created only once - Remove per-test create_all/drop_all operations - Keep transaction rollback for test isolation This ensures: 1. Schema is created once at the start of the test session 2. No race conditions during concurrent test execution 3. Tests remain isolated through transaction rollback https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
Move module-level global variables after all imports to comply with Python style guidelines (PEP 8). https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
The asyncio.Lock was event-loop specific, which caused race conditions when pytest-asyncio creates different event loops for different tests. Multiple tests could bypass the lock and try to create the same database indexes concurrently, causing DuplicateTableError. Changes: - Import threading module - Replace asyncio.Lock() with threading.Lock() - Change async with to regular with for the lock - Add comments explaining the event loop issue threading.Lock works across all event loops within the same process, ensuring only one test creates the database schema regardless of which event loop it runs on. https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
Remove redundant index=True from created_at column to prevent DuplicateTableError. The index is already defined explicitly in __table_args__ as ix_user_read_model_created_at. This fixes 9 test errors in test_concurrency.py where tests failed during database schema creation due to attempting to create the same index twice. https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
Fixed two critical test collection issues:
1. Contract tests (test_api_contract.py):
- Moved schema loading from module level to conditional import
- Added server availability check before schema loading
- Prevents connection errors during test collection
- Tests now skip gracefully when API server is not running
- Uses pytestmark to skip entire module if server unavailable
2. Infrastructure test structure:
- Added missing __init__.py files in test directories:
* tests/unit/infrastructure/__init__.py
* tests/unit/infrastructure/compliance/__init__.py
* tests/unit/infrastructure/repositories/__init__.py
- Fixes ModuleNotFoundError for test_security_extended.py
- Ensures proper Python package structure for pytest
Test results:
- Unit tests: 1873 passed with 86.30% coverage (exceeds 80% requirement)
- Test collection: 2243 tests collected without errors
- All unit tests passing successfully
Benefits:
- Tests can run without requiring external services to be running
- Better CI/CD compatibility
- Clear skip messages when services unavailable
- Proper module structure prevents import errors
https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
Resolved ruff linting error I001 by correcting the import order. https://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements a comprehensive set of enterprise-grade features for the Python Fast Forge framework, including compliance management, extensible plugin system, CQRS pattern with event sourcing, and operational documentation.
Key Changes
Enterprise Compliance Framework
src/infrastructure/compliance/hipaa.py): Encryption, audit logging, access control, and data integrity for ePHIsrc/infrastructure/compliance/gdpr.py): Data subject rights (access, erasure, portability), consent management, breach notificationsrc/infrastructure/compliance/iso27001.py): Security controls for access rights, authentication, cryptography, and monitoringsrc/infrastructure/compliance/soc2.py): Common and additional criteria implementation for service organizationssrc/infrastructure/compliance/manager.py): Unified interface coordinating all compliance frameworksPlugin System
src/infrastructure/plugins/base.py): Protocol-based plugin interface with lifecycle managementsrc/infrastructure/plugins/manager.py): Discovery, loading, and runtime management of pluginssrc/infrastructure/plugins/builtin/auth.py): JWT, OAuth2, SAML, LDAP supportsrc/infrastructure/plugins/builtin/email.py): SMTP, SendGrid, AWS SES, Mailgun providerssrc/infrastructure/plugins/builtin/storage.py): S3, GCS, Azure Blob, local filesystemsrc/app/usecases/plugin_usecases.py): Business logic for plugin managementCQRS & Event Sourcing
src/app/commands/__init__.py,src/app/queries/__init__.py): Separation of write and read operationssrc/app/command_handlers/__init__.py,src/app/query_handlers/__init__.py): Processing logic for commands and queriessrc/domain/events/): Event base classes, user events, and event bus for pub/subsrc/infrastructure/persistence/event_store_models.py): Append-only event log with snapshotssrc/infrastructure/repositories/event_store_repository.py): Persistence layer for eventssrc/infrastructure/persistence/read_models.py): Denormalized tables for optimized queriessrc/infrastructure/projections/user_projection.py): Projection workers to keep read models in syncsrc/app/events/handlers/user_event_handlers.py): Decoupled reactions to domain eventsMessaging & Scheduling
src/infrastructure/messaging/queue.py): Backend-agnostic queue interfacesrc/infrastructure/messaging/rabbitmq.py): AMQP-based message brokersrc/infrastructure/messaging/redis_queue.py): Lightweight Redis-based queuesrc/infrastructure/messaging/scheduler.py): CRON and interval-based task schedulingConfiguration Refactoring
AppSettings: Application and server configurationDatabaseSettings: Database connection and poolingSecuritySettings: JWT, CORS, rate limitingCacheSettings: Redis configurationObservabilitySettings: Telemetry and tracingWorkflowSettings: Temporal configurationPluginSettings: Plugin system configurationExternalServicesSettings: Email, SMS, and external service configurationUtilities & Patterns
src/utils/result.py): Functional error handling withhttps://claude.ai/code/session_011CV2C39yWrAYPJYVPv5Dnv