AEAD encrypt/decrypt: reuse an output buffer instead of malloc/free per chunk

[ThomasWaldmann]

The AEAD encrypt/decrypt paths allocate and free a fresh output buffer for every chunk:

https://github.com/borgbackup/borg/blob/master/src/borg/crypto/low_level.pyx

246:            odata = <unsigned char *>PyMem_Malloc(hlen + self.mac_len + self.iv_len_short + ...)
278:                PyMem_Free(odata)
306:            odata = <unsigned char *>PyMem_Malloc(ilen + self.cipher_blk_len)
335:                PyMem_Free(odata)
517:            odata = <unsigned char *>PyMem_Malloc(hlen + self.mac_len + ...)   # AES-OCB / CHACHA
557:                PyMem_Free(odata)
594:            odata = <unsigned char *>PyMem_Malloc(ilen + self.cipher_blk_len)
630:                PyMem_Free(odata)

Chunks are typically ~2 MiB, so each malloc/free pair is large enough that the allocator hands the pages back to the OS and faults them in again for the next chunk. In a profile this shows up as madvise plus the zeroing/copying around it, once per chunk, on both create and extract.

borg already has the right tool for this: helpers/datastruct.py:Buffer is a managed, resizable, reusable buffer with an optional limit, and compress.pyx uses it for the lz4 path (buffer = Buffer(bytearray, size=0)). The AEAD layer just does not use it.

How much is it worth

Profiled on master (f8dee0c), 6 GiB compressible file, zstd,3 / aes256-ocb / sha256 ids, default (fastcdc) chunker, macOS/arm64, native sampling, idle samples excluded:

	create	extract
memory management, total	11.6% of cpu	16.8% of cpu
of which AEAD output buffer alloc/free	1.9%	1.9%

So roughly 2% of cpu on each path, which is the single largest borg-owned item inside the memory-management bucket. (The largest item overall in extract is zstd's own output assembly at ~6%, but that is inside the Python zstd binding's _OutputBuffer and the stdlib API has no decompress_into, so it is not ours to fix.)

For context, the full memory-management breakdown for create, as % of total cpu:

 1.74%  memoryview release
 1.69%  AEAD encrypt: free output buffer      <-- this issue
 1.48%  PyBuffer_ToContiguous -> bytes()
 1.04%  bytearray setslice/extend
 0.99%  FileReader.read: free buffer
 0.95%  zstd _OutputBuffer list free
 0.92%  fastcdc: PyBytes per chunk
 0.63%  ZSTD_freeCCtx (fresh ctx per call)
 0.34%  PackWriter b"".join
 0.18%  repoobj hdr+meta+data concat

and extract:

 3.70%  zstd output assembly (join blocks)    (in the zstd binding, not ours)
 3.12%  ZSTD_decompressStream copy            (inside libzstd)
 2.35%  zstd _OutputBuffer list free          (in the zstd binding)
 2.23%  frame clear / misc
 1.33%  bytes_subscript (PackReader slice)    -> #10030
 1.21%  ZSTD_freeDCtx (fresh ctx per call)
 1.19%  AEAD decrypt: free output buffer      <-- this issue
 0.69%  AEAD decrypt output                   <-- this issue

Notes / caveats

• This is not a single big win. Memory management is 12-17% of cpu, but it is spread over a dozen sites of 1-2% each. Reusing the AEAD buffer plus PackReader.read(): return a memoryview instead of copying out of the in-memory pack #10030 is worth maybe 3-4% per path, not more. Filing it because it is cheap and the pattern already exists in the codebase.
• A reused buffer has to be per-key-instance (or otherwise not shared across threads) if borg ever grows chunk-level parallelism - worth keeping in mind given Multithreading #37 / multithreading: input file discovery / reading parallelism #3500.
• Buffer grows monotonically and never shrinks, so a limit should be set; the natural bound is MAX_DATA_SIZE plus the AEAD overhead.
• Two related-but-separate observations from the same profile, not worth their own issues yet: a fresh ZstdCompressor/ZstdDecompressor context is created per chunk (~0.6% create / ~1.2% extract - I measured reuse and it gives no throughput gain at 2 MiB chunks, only removes the churn), and PyBuffer_ToContiguous -> bytes() costs 1.48% of create from a call site I have not pinned down yet.
• All numbers are from one machine (Apple M3 Pro, 12 cores) with one workload shape. On already-compressed data the profile shifts a lot, since zstd bails out early and everything else grows proportionally.

🤖 Generated with Claude Code