[FLINK-40296][core] Add object-level migrate hook to TypeSerializerSnapshot - #28880
Open
weiqingy wants to merge 1 commit into
Open
[FLINK-40296][core] Add object-level migrate hook to TypeSerializerSnapshot#28880weiqingy wants to merge 1 commit into
weiqingy wants to merge 1 commit into
Conversation
…apshot
Add a default method that lets a serializer snapshot transform an already
deserialized state value from the schema it was written with into the schema
the current serializer expects:
default T migrate(TypeSerializerSnapshot<T> oldSerializerSnapshot, T value)
Like resolveSchemaCompatibility, it is invoked on the new snapshot and receives
the old snapshot as its argument. The default returns the value unchanged, so
behavior is unaffected for every existing serializer: a value deserialized with
the prior serializer is structurally compatible with the current one and can be
re-serialized as is.
The javadoc states that migration is not applied recursively to nested
serializers. Unlike resolveSchemaCompatibility, which CompositeTypeSerializer-
Snapshot delegates to the nested snapshots, migrate has no delegating override,
so a composite returns its value unmigrated unless it decomposes the value
itself. That asymmetry is invisible at the call site and would otherwise fail
silently.
Generated-by: Claude Code (Opus 5)
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the first PR of the FLIP-527 implementation, split into a stack of small, independently reviewable PRs under the umbrella issue FLINK-37732. Landing order:
migratehook onTypeSerializerSnapshotRowDataEach PR depends on the one before it. PR-1 and PR-2 are behavior-neutral: the hook added here defaults to returning its argument, so no existing serializer changes behavior until PR-3 overrides it for
RowData.What is the purpose of the change
FLIP-527 makes a
RowDatastate value survive a backward-compatible schema change, such as a nullable field appended by an upstream Avro schema. Doing that needs a way for a serializer snapshot to transform an already deserialized value from the schema it was written with into the schema the current serializer expects.RowDataneeds this because its binary layout is fixed-width and position-addressed, so re-serializing a value read with the old layout does not re-pack it.This PR adds only that extension point:
Like
resolveSchemaCompatibility, it is invoked on the new snapshot and receives the old snapshot as its argument. The default returns the value unchanged, which is correct for every serializer whose in-memory representation does not depend on the schema: such a value is structurally compatible with the current serializer and can be re-serialized as is.Nothing calls the hook yet. The caller lands in PR-2 and the first override in PR-3. Keeping the interface change on its own keeps the public extension point reviewable in isolation.
Brief change log
migratemethod toTypeSerializerSnapshot, returning the value unchangedresolveSchemaCompatibility, whichCompositeTypeSerializerSnapshotdelegates to the nested snapshots,migratehas no delegating override, so a composite returns its value unmigrated unless it decomposes the value itselfVerifying this change
This change added tests and can be verified as follows:
TypeSerializerSnapshotTest#testMigrateReturnsValueUnchangedByDefaultasserts that a snapshot which does not overridemigratereturns the same instance it was given. The identity assertion is deliberate rather than an equality one: a default that copied would be a silent per value allocation on the restore path.Does this pull request potentially affect one of the following parts:
@Public(Evolving): yes,TypeSerializerSnapshotis@PublicEvolving. The addition is adefaultmethod, so it is source and binary compatible and no existing implementor needs to change.Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5)