[FLINK-39900] [forst] Add configuration option for setting the number of ForSt transfer threads

[francis-a]

What is the purpose of the change

Currently the number of threads used by the ForStStateDataTransfer class is hardcoded to 4. This PR proposes a new state.backend.forst.checkpoint.transfer.thread.num configuration option that allows for providing a custom thread number value.

Brief change log

• Added a new state.backend.forst.checkpoint.transfer.thread.num config option
• Updated the ForStResourceContainer with a getter to return this option value
• Updated ForStStateDataTransfer to provide the new config option value instead of the hardcoded default of 4
• Removed the hardcoded default

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): / no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? Documentation has been added using the existing ConfigOption class

[flinkbot]

CI report:

• 930d4de Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[spuru9]

LGTM
Can you run mvn spotless:apply

[davidradl]

This we should document the default value of 4. Also there should be documentation to describe bet practises around when this new option should be used and any side effects if it is incorrectly configured.

[francis-a]

Thanks, I updated this to be more explicit.

[Zakelly]

Thanks, LGTM

[snuyanzin]

@Zakelly , @spuru9 , @francis-a this commit broke the build

[snuyanzin]

here it is the fix #28859
can someone please approve?

[snuyanzin]

everything is much worse
the test is expecting 10 and actual is 4
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=77624&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=1ffc5ec2-7913-50ff-0177-3fca16f1b8f0&l=12254

I'm going just revert the problem commit (fyi: @spuru9 @Zakelly @francis-a )

UPD:
Moreover I checked CI for this PR and it is failing on exactly same test (introduced in this PR) with exactly same issue https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=77352&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=1ffc5ec2-7913-50ff-0177-3fca16f1b8f0&l=11970

So I don't understand why this PR then was merged ?

Please do NOT merge code which does NOT pass CI tests first

[Zakelly]

@snuyanzin oh sorry, my fault. I was interrupted halfway through the review and mistakenly thought the CI for this PR had already passed.

Thanks for the revert.

[francis-a]

@snuyanzin @Zakelly sorry about that, that is on me for not noticing the failing test. Thanks for taking care of the revert. I will re-open this change set with the needed test fixes.

[francis-a]

The new PR is open here: #28897

The test failure caught a real issue with the new property missing from the config merge set in ForStConfigurableOptions