Skip to content

Process replication slot changes - #4930

Open
DilwoarH wants to merge 26 commits into
mainfrom
process-replication-slot-changes
Open

Process replication slot changes#4930
DilwoarH wants to merge 26 commits into
mainfrom
process-replication-slot-changes

Conversation

@DilwoarH

@DilwoarH DilwoarH commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

** DO NOT MERGE UNTIL #4946 IS MERGED IN FIRST**

What

The application code for processing slot changes for notifications table

Why

This is needed to process WAL logs in the database

Test results

Ran a load test on dev-c - this is the result after 2400 emails.

SELECT
	(SELECT count(*) FROM pg_logical_slot_peek_changes('notify_dashboard_replication_slot', NULL, NULL)) AS still_processing,
    (SELECT SUM(notification_count) FROM ft_service_stats) AS ft_stats_count,
    COUNT(*) AS notif_count,
    (SELECT SUM(notification_count) FROM ft_service_stats where notification_status = 'created') AS ft_created_count,
    COUNT(CASE WHEN notification_status = 'created' THEN 1 END) AS notif_created_count,
    (SELECT SUM(notification_count) FROM ft_service_stats where notification_status = 'sending') AS ft_sending_count,
    COUNT(CASE WHEN notification_status = 'sending' THEN 1 END) AS notif_sending_count,
    (SELECT SUM(notification_count) FROM ft_service_stats where notification_status = 'delivered') AS ft_delivered_count,
    COUNT(CASE WHEN notification_status = 'delivered' THEN 1 END) AS notif_delivered_count,
    (SELECT SUM(notification_count) FROM ft_service_stats where notification_status = 'temporary-failure') AS ft_temp_fail_count,
    COUNT(CASE WHEN notification_status = 'temporary-failure' THEN 1 END) AS notif_temp_fail_count,
    (SELECT SUM(notification_count) FROM ft_service_stats where notification_status = 'permanent-failure') AS ft_perm_fail_count,
    COUNT(CASE WHEN notification_status = 'permanent-failure' THEN 1 END) AS notif_perm_fail_count
FROM
    public.notifications;
still_processing ft_stats_count notif_count ft_created_count notif_created_count ft_sending_count notif_sending_count ft_delivered_count notif_delivered_count ft_temp_fail_count notif_temp_fail_count ft_perm_fail_count notif_perm_fail_count
0 2400 2400 0 0 0 0 1206 1206 234 234 960 960

How it works

605101533-57f970e2-60d7-462a-9fbd-b24d553d6663

@DilwoarH
DilwoarH force-pushed the process-replication-slot-changes branch 6 times, most recently from 6bddcf6 to 2d6790a Compare August 11, 2026 09:53
This will be replaced with the actual task in the following commits
the class is not processing all replication slot changes - just notifications
Also include bst_date in the dimensions
@DilwoarH
DilwoarH force-pushed the process-replication-slot-changes branch from ed72e86 to 35de7dd Compare August 17, 2026 22:20
@DilwoarH
DilwoarH force-pushed the process-replication-slot-changes branch from da47acd to 1f988ae Compare August 18, 2026 00:15
@DilwoarH DilwoarH changed the title [WIP] Process replication slot changes Process replication slot changes Aug 18, 2026
@DilwoarH

Copy link
Copy Markdown
Contributor Author

requires #4936 to be merged first

@DilwoarH
DilwoarH force-pushed the process-replication-slot-changes branch from 1f988ae to 1d032b9 Compare August 18, 2026 00:29
"notification_type": dimensions["notification_type"],
"notification_status": dimensions["notification_status"],
}
filters = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filters is used only in else block so better to move inside else block to avoid evaluating the SQLAlchemy expressions on increments

change_counts: Counter[ServiceStatsDimensionsKey] = Counter()
for dimensions, change_count in counter.items():
bst_date, template_id, service_id, notification_type, notification_status = dimensions
change_counts[(bst_date, service_id, template_id, notification_type, notification_status)] += change_count

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a confusing order swap between template_id and service_id, is this intentional?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no - good spot

if key_type == "test":
return None

if require_status_from_primary_row and not notification_status:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need this condition check? Updated notifications always have a previous notification_status from the database, this condition seems redundant and a bit confusing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants