[FLINK-38774][table-common] Skip hidden directories when recursively listing partitions

[hulincup]

What

PartitionPathUtils.listStatusRecursively descended into every subdirectory returned by fs.listStatus, including hidden ones such as _temporary. A non-hidden child of a hidden dir (e.g. _temporary/job-123) sitting at partition depth was then collected with an empty partition spec, which later surfaces as a TableException: incomplete partition spec in the filesystem connector source.

Reported in FLINK-38774 (supersedes the older FLINK-31975).

Fix

Reuse the existing listStatusWithoutHidden helper to filter hidden children at every recursion level — not only at the leaf, where searchPartSpecAndPaths already filtered via isHiddenFile. Add a null guard consistent with listStatusWithoutHidden's contract (returns null when fs.listStatus returns null), which also removes a latent NPE in the old for-each over a null list.

if (fileStatus.isDir()) {
    FileStatus[] children = listStatusWithoutHidden(fs, fileStatus.getPath());
    if (children == null) {
        return;
    }
    for (FileStatus stat : children) {
        listStatusRecursively(fs, stat, level + 1, expectLevel, results);
    }
}

Test

Adds a @TempDir regression test that creates a real partition (date=2019-8-30/country=China) alongside a hidden _temporary/job-123 subtree and asserts searchPartSpecAndPaths returns only the real partition. Without the fix the hidden job-123 leaf leaks in with an empty spec (2 entries); with the fix only the real partition is returned (1 entry).

Notes

• Supersedes the abandoned [FLINK-38774] Fixed incorrect handling of hidden directories when obtFixed incorrect handling of hidden directories when obtaining partition structureaining partiti… #27314 (stale-closed after 120 days of inactivity, no technical objection — reviewer only asked for a unit test, which is included here).
• This is a flink-table/flink-table-common utility, not literally under flink-connectors/, but the bug manifests via the filesystem connector (tagged Connectors / FileSystem).

[flinkbot]

CI report:

• 8bb6c43 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build