Skip to content

fix(worker): use stored block length for replication reads (#1634) - #1637

Open
gangump82 wants to merge 1 commit into
CurvineIO:mainfrom
gangump82:fix/1634-replication-reader-length
Open

fix(worker): use stored block length for replication reads (#1634)#1637
gangump82 wants to merge 1 commit into
CurvineIO:mainfrom
gangump82:fix/1634-replication-reader-length

Conversation

@gangump82

@gangump82 gangump82 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix block replication failures caused by opening the source block reader with a
zero logical length.

Replication now derives its read boundary from the finalized block metadata
while preserving the explicit logical-length behavior used by normal client
reads and sparse/truncated blocks.

Issue Describe / Design

Closes #1634

Root cause

WorkerReplicationManager::replicate_block opened the source block with:

open_reader_by_id(job.block_id, 0, 0)

The third argument is the reader's logical length. After the strict logical
boundary behavior introduced for sparse and truncated block reads, zero is a
valid zero-length boundary rather than a request to infer the physical block
length.

As a result, the first replication read_region call failed with:

offset exceeds block length, length=0, offset=0

The error happened while reading the source replica, before data could be
written to the target Worker.

Reproduction

  1. Start a cluster with at least three Workers and block replication enabled.
  2. Write a non-empty block with fewer replicas than the configured target.
  3. Allow the Master to schedule replication.
  4. The source Worker opens the block reader with logical length zero.
  5. The first source read fails and the replica count does not increase.

Fix approach

Add a dedicated BlockStore::open_reader_by_id_at_stored_len API for internal
block-copy operations that do not have a client-provided logical length.

The API obtains the current readable BlockMeta, derives the logical boundary
from BlockMeta::len(), resolves the generation and layout, and opens the file
descriptor while holding the same dataset read lock.

The existing explicit-length API remains unchanged. In particular, this change
does not restore a max(logical_len, physical_len) fallback, so sparse and
truncated reads retain their exact logical boundaries.

Changes

Module / File Change Impact on existing behavior
curvine-worker/src/worker/block/block_store.rs Add a stored-metadata-length reader API and share reader creation through an internal helper Internal block copies can infer the correct logical length; explicit client read lengths are unchanged
curvine-worker/src/worker/block/block_store.rs Add a regression test for reading a finalized non-empty block using its stored length Test-only
curvine-worker/src/worker/block/block_store.rs Exercise the new API in the committed-generation reader test Test-only
curvine-worker/src/worker/replication/worker_replication_manager.rs Open replication source readers using the finalized block metadata length Replication can copy non-empty blocks instead of failing with a zero-length read boundary

Test verified

Test case Result Notes
cargo test -p curvine-worker PASS 31 passed, 0 failed, 1 ignored
cargo test -p curvine-tests --test replication_test test_replication_honors_source_block_capacity -- --nocapture PASS Replication completed without length=0; source capacity was preserved
cargo test -p curvine-tests --test replication_test test_block_replication_e2e -- --nocapture PASS Replica count increased from 2 to 3 and 204800 bytes passed integrity verification
cargo fmt --all -- --check PASS
git diff --check PASS

Dependencies

@szbr00
szbr00 requested a lite review from Copilot August 28, 2026 02:16
@gangump82
gangump82 marked this pull request as ready for review August 28, 2026 02:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes Curvine worker block replication failures by ensuring the source reader uses the finalized block’s stored logical length (instead of an explicit zero), aligning replication reads with the strict logical-boundary semantics used elsewhere.

Changes:

  • Add BlockStore::open_reader_by_id_at_stored_len for internal read paths that should derive the logical boundary from stored BlockMeta::len().
  • Refactor open_reader_by_id to share the dataset-lock-held reader-open logic via an internal helper.
  • Update WorkerReplicationManager::replicate_block to open replication source readers using the stored metadata length, and add/adjust unit tests to cover the new API.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
curvine-worker/src/worker/replication/worker_replication_manager.rs Use a stored-metadata-length reader for replication source reads to avoid zero-length logical boundaries.
curvine-worker/src/worker/block/block_store.rs Introduce a stored-length reader API, factor common reader-open logic, and add regression coverage validating finalized reads via stored length.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Block replication remote write fails on zero-length staging block

2 participants