A practical implementation of multiple reliable event-driven design patterns to solve the Dual Write Problem using Kafka, PostgreSQL, Debezium, and Microservices Architecture.
In distributed systems and microservice architectures, services often need to perform two operations together:
- Write data to the database
- Publish an event/message to a message broker like Kafka
This creates the Dual Write Problem.
The main challenge with dual writes is maintaining consistency between the database and the message broker. Since both operations happen independently, failures during either operation can lead to inconsistent system states, such as:
- Data being stored in the database but the event not being published
- Event being published but the database transaction failing
This repository demonstrates different approaches to solving the Dual Write Problem using reliable event-driven design patterns.
The service writes both the business data and an outbox event into the same database transaction. A separate process then reads the outbox table and publishes events to Kafka, ensuring reliable event delivery and maintaining consistency.
The service publishes events to Kafka and also consumes its own events to update internal state or trigger further processing. This pattern enables event-driven workflows while helping maintain consistency across services.
Instead of directly publishing events, database transaction logs (WAL/binlogs) are monitored using tools like Debezium. Changes are captured directly from the database logs and streamed to Kafka, ensuring reliable and consistent event publishing.
This project aims to demonstrate and simulate how modern distributed systems solve consistency problems between database operations and asynchronous event publishing using different architectural patterns and messaging strategies.
Below is the database schema used in the project architecture.
Before starting the project, make sure Docker and Docker Compose are installed on your system.
Go into each service directory and build the containers using Docker Compose.
docker compose up --build -d
Repeat this step for all the required services/containers in the project.
After all Docker containers are created successfully, move to the root directory of the project and run:
chmod +x start_all_services.sh
./start_all_services.sh
This script will automatically start all required backend services, pollers, consumers, and supporting infrastructure.
After the Debezium container starts successfully, attach the connector manually during installation.
Example:
curl -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" \
-d @connector.json
Make sure:
-
Kafka is running
-
Kafka Connect is running
-
PostgreSQL/MySQL database is accessible
-
connector.jsoncontains the correct database configuration
You can verify the connector using:
curl http://localhost:8083/connectors
To check whether all containers are running:
docker ps
To check logs of a specific container:
docker logs <container_name>
To stop all running containers:
docker compose down
Orders_1___Transactional_Outbox_PatternOrders_2___Listen_To_Yourself_PatternOrders_3___Transactional_Log_Tailing
order_service_1-backend-1order_service_2-backend-1
These services are responsible for:
- Writing order data to PostgreSQL
- Publishing or generating events for Kafka
- Demonstrating different solutions to the Dual Write Problem
postgres_outbox
Stores business order data for all implemented patterns.
Pattern Types
Transactional_OutboxListen_To_YourselfTransactional_Log_Tailing
Stores events/messages for the Transactional Outbox Pattern before they are published to Kafka.
Stores events/messages used in the Listen To Yourself Pattern workflow.
Used for idempotency in the Listen To Yourself Pattern to prevent duplicate event processing.
Schema and table structures can be explored using Prisma Studio.
Pollers continuously monitor outbox tables and publish pending events to Kafka.
poller_1_transactional_outbox-backend-1poller_2_transactional_outbox-backend-1
poller_1_listen_to_yourself-backend-1poller_2_listen_to_yourself-backend-1
Debezium is used to implement the Transactional Log Tailing Pattern by capturing database changes directly from PostgreSQL transaction logs and streaming them to Kafka.
curl -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" \
--data @register-postgres.jsoncurl -X DELETE http://localhost:8083/connectors/postgres-connectorcurl http://localhost:8083/connectorsThe integration between Debezium and Kafka is configured inside:
docker-compose.yml
Client Request
↓
Order Service
↓
PostgreSQL Database
↓
┌───────────────────────────────┐
│ Dual Write Solution Patterns │
└───────────────────────────────┘
↓
├── Transactional Outbox Poller
├── Listen To Yourself Poller
└── Debezium CDC
↓
Kafka Topics
↓
Consumers / Downstream Services
⭐ Feel free to fork, improve, and add your own innovations to this project. If you found this repository useful, consider giving it a star!