Implement Batching for Performance

[mgazza]

Overview

Improve existing batching mechanism with configurable limits and performance monitoring.

Current State

We already have essential batching for sync operations:

• GetLatestChanges: Caps at 100k changes total (prevents memory/network overload)
• MergeChanges: Processes in 10k transaction batches (avoids DB transaction size limits)

Improvements Needed

• Make batch sizes configurable instead of hardcoded constants
• Add batch performance metrics - track batch utilization and timing
• Evaluate optimal batch sizes - are 100k/10k appropriate defaults?
• Consider adaptive batching - adjust sizes based on system performance
• Add batch size validation - ensure configured values are reasonable

Priority

Phase 2: Performance at Scale - Optimize existing foundation

Labels

enhancement, performance, batching

[cmsegal]

We do have a very simple batching mechanism in place for sync operations.
To give you an overview of what we currently have:
In GetLatestChanges (method that retrieves recent changes requested by another peer), we fetch a batch of up to 100k changes, but this number should probably be reduced:

clset-example/crdt.go

Line 314 in f419dcb

if count == 100000 {

In MergeChanges (method that merges an incoming list of changes), we merge the changes in batches (each one a DB transaction) of 10k:

clset-example/crdt.go

Line 349 in f419dcb

func (c *CRDT) MergeChanges(fromPeerID string, changes []KeyEntry, tracked map[string]uint64) error {

These batch sizes are currently hardcoded.
Can you have a look and see what should be added or changed / improved / replaced in this existing mechanism?

[mgazza]

Thanks for the context! After reviewing the code, I can see there's already solid batching infrastructure:

GetLatestChanges (crdt.go:314): Caps at 100k changes total when collecting from all peers - prevents memory/network issues during large syncs.

MergeChanges (crdt.go:349): Processes incoming changes in 10k transaction batches - smart database transaction management to avoid size limits.

This is essential infrastructure for handling large datasets safely, not just performance optimization.

Potential improvements:

1. Make batch sizes configurable - different deployments might need different limits
2. Add metrics - visibility into batch utilization and performance
3. Evaluate optimal sizes - are 100k/10k the right defaults?