[DO NOT MERGE] Pending bug fixes, for review

[ayzk]

DO NOT MERGE — 应 @ayzk 要求,此 PR 仅供审阅,不要合并。

master 上先前的合并已被撤销(4ac704f4),master 的内容与 57ce9e90 完全一致。本分支保留全部内容待逐项审阅。

新增一个提交(06db6065):本树自己分配的四处 scratch buffer 改为 unique_ptr<uchar[]>。原先是 malloc 之后交给带 free() deleter 的 unique_ptr,留了一个裸指针别名在作用域里,且四处都没检查分配结果。new uchar[] 对 uchar 是 default-initialize(不清零 —— 实测 1.12 GB 下 make_unique<uchar[]> 要 151 ms,new uchar[] 0 ms),失败时抛而不是返回 null。需要裸起始指针的地方用 const 别名而不是到处 .get()。SZGenericCompressor::decompress 那处保留 free() deleter,因为那块缓冲是 lossless 层分配的。

Bug fixes only. Nothing here changes the compressed format or an interface signature, and every algorithm's output is byte-identical to master's -- verified over 31 combinations of dataset (miranda velocityx.d64 float64, cesm-atm ODV_bcar1 float32, the in-tree smoke file), algorithm and error bound, plus decompressing files master itself produced.

Supersedes #143.

The open pull requests, reviewed line by line

PR	Defect	Taken
#131	Config::load computed the end of the config blob as c + confSize after c had already advanced past the length prefix, one byte too far. save() includes the prefix in confSize, so the end is c0 + confSize. The last optional-field guard then fires with one byte left and reads a field that is not there -- past the end of the buffer, since the config is the last thing in the stream.	yes (via #132's rewrite, which contains the same fix)
#132	Bounds-checks the whole decompression path: header size in SZ_decompress (which previously ignored the cmpSize it was given entirely), a bounded Config::load overload with N/bit-width/dims-product validation, Lossless_zstd (including returning a zstd error code as a size), Huffman tree loading, unpad_tree child indices, predictor selection indices, recover_unpred, unpred_size before resize, OMP thread counts and per-thread sizes, and RAII for buffers freed on the throwing paths.	yes, except two parts -- see below
#133	out1 << (64 - len) with len == 0 -- a single-symbol Huffman tree, which a constant field produces -- shifts a 64-bit value by 64.	yes
#134	HuffmanEncoderV2::loadAsDFSOrder read a fixed header and a DFS bitstream with no bounds at all, reserve(tree.n << 1) overflowed, and it advanced the cursor without ever decrementing remaining_length.	yes, plus two gaps closed below
#135	smallIdx, read from the compressed stream, indexed the fixed-size magicInts table unchecked; the first malloc in XtcBasedEncoder was overwritten and leaked; Lossless_bypass did not check its malloc.	yes
#137	RegressionPredictor::load debited remaining_length by the decoded bin count while decode() advances by the much smaller encoded byte count, so the budget ran out early; ComposedPredictor::load had the mirror-image bug and never debited it at all, leaving the bound too loose.	yes
#138	static_cast<int64_t>(fabs(diff) * error_bound_reciprocal) is UB for NaN, infinities and huge magnitudes.	yes -- observable behaviour is unchanged (NaN/inf/1e38 produce the same bin and the same written-back value before and after)
#139	Scratch buffers were free()d at the end of a function that can throw; and the OMP path's per-chunk capacity omitted the size header Lossless_zstd::compress writes, so a poorly compressible chunk threw.	yes

The two parts of #132 not taken

The compressed format change. #132 moves the quant_inds count from after the encoder's serialized tree to before it, so the tree is immediately followed by its encoded stream and the bound load() records is exact. That is a format change and SZ3_DATA_VERSION is not bumped, so the version check passes and a file written by any released SZ3 fails with a misleading SZ3 Huffman: invalid node count. Verified by decompressing a master-produced file with #132's build.

The bound it was buying is recovered without touching the format, by letting the caller supply it instead of having load() infer it:

// HuffmanEncoder
void set_decode_bound(size_t remaining);

SZGenericCompressor calls it once it has consumed the count field, and both predictors call it because their tree and stream are contiguous. load() no longer records a bound at all: the tree and the encoded stream are not required to share a buffer, and tools/test/modules/test_encoder.cpp deliberately puts them in separate ones -- inferring the bound there made that test fail.

The dstLen reinterpretation. #132 treats LosslessInterface::decompress's dstLen as an input capacity when the caller supplies the buffer. The declared contract is that it is an output, and callers do pass uninitialised values -- LosslessTest.LosslessBypass fails with it, and LosslessTest.LosslessZstd only passed because the garbage happened to be large. The self-allocating branch, where a non-zero incoming value is opt-in and zero means no bound, is kept.

Gaps found while reviewing, fixed here

• HuffmanEncoderV2 sizes its dense tables with veccode.resize(tree.maxval), and maxval is read straight from the stream. Bounds-check HuffmanEncoderV2 tree loading against corrupted input #134 bounds the node count but not this, leaving an unbounded allocation. Bounded by the encoder's own invariant (preprocess_encode only leaves usemp == 0 while maxval < 1 << 28).
• tree.n is an int holding a value read as 64-bit, so Bounds-check HuffmanEncoderV2 tree loading against corrupted input #134's new bound is checked with an explicit sign test rather than letting the comparison convert it (which also silenced a -Wsign-compare warning the PR introduced).
• tools/test/modules/test_lossless.cpp passed an uninitialised size to decompress().
• Nothing in the suite exercised the HDF5 filter's buffer sizing: a chunk small enough for the
  compressed block to exceed the raw chunk size never appeared. test_h5_filter.py gains a third
  chunk mode that asks for 8-element chunks, which the pre-fix filter rejects with
  buffer not large enough on every algorithm. The mode is restricted to a leading slice of at
  most 4096 chunks -- over a whole 280M-element field it would mean 35M chunks, and the HDF5 chunk
  index alone takes the process to 15.3 GiB of a runner's 16 GiB. Measured in a 16 GiB container at
  hacc's exact size: 9.6 GiB and passing.

From the fz branch, verified to reproduce here

• Lossless_bypass::compress ignored its destination capacity and memcpy'd the payload regardless -- an unconditional heap overflow whenever the payload is larger than the caller's buffer. ALGO_BIOMDXTC is the one algorithm that pairs its codec with bypass, so nothing shrinks the payload; AddressSanitizer on Linux caught it as a 59599-byte overflow, glibc reported munmap_chunk(): invalid pointer, and macOS's allocator did not abort at all.
• TimeSeriesDecomposition violated its error bound 1.94x with data_ts0 == nullptr: block_data's compress-side path has no write-back, so timestep 0 was never updated with its reconstruction while decompress() predicted from its own. 0.997x after the fix.
• ArithmeticEncoder sign-extended bytesToInt64_bigEndian(bytes) >> 20, pushing the value outside the 44-bit MAX_CODE window. A 60-stream sweep gave 30 correct, 23 wrong, 7 SIGSEGV.
• HuffmanEncoder's stateNum = max - offset + 2 narrows to int, so a wide bin range goes negative and the state-table malloc is UB.
• RunlengthEncoder, NoPredictionDecomposition and InterpolationDecomposition had no size_est(), so SZGenericCompressor sized its buffer from 0 while the quantizer's unpredictable list could be arbitrarily large.
• KmeansUtil had reserve() followed by operator[] writes, and uniform_int_distribution(0, num) indexing data[num]. Live in mdz when dims[1] > 5000.
• The HDF5 filter sized its compressed buffer below SZ_compress's own minimum, so an explicit chunk of a few hundred elements aborted.
• 24 headers were not self-contained under one or both standard libraries. libc++ provides more transitively than libstdc++, so this is latent until a translation unit changes its include order -- it broke the Linux build on fz once a new test file did.
• .gitignore's bare test pattern matched tools/test/, hiding every test file from git.

Verification

• clang and GCC 16, Release + BUILD_TESTING=ON + BUILD_H5Z_FILTER=ON: 0 errors, 0 warnings, all tests pass
• Every header compiles standalone under both toolchains (57/57)
• 31 dataset x algorithm x error-bound combinations byte-identical to master; every master-produced file decompresses to the same bytes
• Each ported fix was A/B'd against its unfixed state

Generated with Claude Code